Oct 31 07:25:29.876331 ip-10-0-140-236 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 07:25:29.876342 ip-10-0-140-236 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 07:25:29.876350 ip-10-0-140-236 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 07:25:29.876585 ip-10-0-140-236 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 07:25:41.170916 ip-10-0-140-236 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 07:25:41.170937 ip-10-0-140-236 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dc38b39266b24898879569410b88ea50 --
Oct 31 07:27:45.074663 ip-10-0-140-236 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 07:27:45.519358 ip-10-0-140-236 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:27:45.519358 ip-10-0-140-236 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 07:27:45.519358 ip-10-0-140-236 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:27:45.519358 ip-10-0-140-236 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:27:45.525885 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525800    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:27:45.525885 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525885    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:27:45.525885 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525888    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:27:45.525885 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525892    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525897    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525900    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525902    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525905    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525908    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525910    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525913    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525916    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525919    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525921    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525924    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525927    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525929    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525932    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525934    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525937    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525940    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525942    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525951    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:27:45.526019 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525953    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525956    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525959    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525961    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525964    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525967    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525969    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525972    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525975    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525977    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525980    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525982    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525985    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525987    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525990    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525993    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525996    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.525998    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526001    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526003    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:27:45.526504 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526006    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526009    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526011    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526015    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526018    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526020    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526023    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526026    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526029    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526031    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526034    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526036    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526039    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526041    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526044    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526046    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526049    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526051    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526054    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:27:45.526965 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526056    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526059    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526061    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526064    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526066    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526069    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526072    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526075    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526078    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526081    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526083    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526086    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526089    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526091    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526094    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526099    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526104    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526109    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526113    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:27:45.527415 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526116    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526119    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526123    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526128    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526131    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526134    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526138    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526140    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526143    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526146    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526149    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526152    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526155    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526157    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526160    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526162    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526165    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526168    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526171    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:27:45.527863 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526173    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526176    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526179    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526182    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526184    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.526188    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527024    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527032    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527035    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527039    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527041    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527044    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527047    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527050    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527053    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527056    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527058    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527061    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527064    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:27:45.528322 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527067    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527069    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527072    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527075    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527078    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527081    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527083    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527086    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527088    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527091    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527094    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527097    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527100    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527102    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527106    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527108    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527111    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527114    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527116    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527119    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:27:45.528833 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527121    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527124    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527127    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527129    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527132    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527134    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527137    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527140    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527143    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527146    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527148    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527152    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527154    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527156    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527159    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527162    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527165    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527167    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527170    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:27:45.529325 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527174    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527176    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527179    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527183    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527187    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527190    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527193    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527196    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527198    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527201    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527203    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527206    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527209    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527211    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527213    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527216    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527218    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527221    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527223    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527226    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:27:45.529769 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527230    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527232    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527235    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527238    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527240    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527243    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527245    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527248    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527250    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527254    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527258    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527261    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527264    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527267    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527270    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527273    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527275    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527296    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527299    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:27:45.530236 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527302    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527304    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527307    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527310    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527312    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527315    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527318    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527321    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527324    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527326    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527329    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527331    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527334    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527337    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527340    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527852    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527858    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527861    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527864    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527867    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:27:45.530716 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527870    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527873    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527876    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527879    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527881    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527884    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527887    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527890    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527892    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527895    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527898    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527900    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527903    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527907    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527910    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527913    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527916    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527919    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527921    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:27:45.531194 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527924    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527926    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527929    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527932    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527934    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527937    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527939    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527943    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527945    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527948    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527951    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527954    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527956    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527959    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527962    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527965    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527968    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527970    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527973    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:27:45.531651 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527975    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527978    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527980    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527983    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527985    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527988    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527990    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527993    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527996    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.527998    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528001    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528003    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528006    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528009    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528011    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528014    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528016    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528019    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528021    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528024    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:27:45.532083 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528026    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528029    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528032    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528034    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528036    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528040    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528045    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528048    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528050    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528053    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528055    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528058    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528060    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528063    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528065    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528068    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528070    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528072    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528075    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528077    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:27:45.532575 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528080    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528084    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528087    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528090    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528092    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528094    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528097    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528099    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528102    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528104    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528107    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528109    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528112    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528115    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528505    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528510    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528513    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528516    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528520    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528523    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:27:45.533050 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528526    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528529    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.528531    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530005    2577 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530016    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530023    2577 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530028    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530033    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530036    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530041    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530045    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530049    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530052    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530057    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530060    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530063    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530066    2577 flags.go:64] FLAG: --cgroup-root=""
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530069    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530072    2577 flags.go:64] FLAG: --client-ca-file=""
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530075    2577 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530078    2577 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530083    2577 flags.go:64] FLAG: --cluster-domain=""
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530086    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530090    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 07:27:45.533588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530093    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530096    2577 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530100    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530104    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530108    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530111    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530115    2577 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530118    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530121    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530124    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530127    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530132    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530135    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530138    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530141    2577 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530144    2577 flags.go:64] FLAG: --enable-server="true"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530147    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530151    2577 flags.go:64] FLAG: --event-burst="100"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530154    2577 flags.go:64] FLAG: --event-qps="50"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530158    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530161    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530164    2577 flags.go:64] FLAG: --eviction-hard=""
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530169    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530172    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530175    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 07:27:45.534148 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530178    2577 flags.go:64] FLAG: --eviction-soft=""
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530181    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530184    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530187    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530190    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530192    2577 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530195    2577 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530198    2577 flags.go:64] FLAG: --feature-gates=""
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530202    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530205    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530209    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530212    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530215    2577 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530219    2577 flags.go:64] FLAG: --help="false"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530221    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-140-236.ec2.internal"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530225    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530228    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530231    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530234    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530238    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530241    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530245    2577 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530248    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 07:27:45.534775 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530251    2577 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530254    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530257    2577 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530276    2577 flags.go:64] FLAG: --kube-reserved=""
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530294    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530299    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530304    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530308    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530311    2577 flags.go:64] FLAG: --lock-file=""
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530314    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530318    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530322    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530328    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530331    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530334    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530337    2577 flags.go:64] FLAG: --logging-format="text"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530340    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530343    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530346    2577 flags.go:64] FLAG: --manifest-url=""
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530349    2577 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530354    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530357    2577 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530361    2577 flags.go:64] FLAG: --max-pods="110"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530364    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530367    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 07:27:45.535322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530370    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530373    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530376    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530387    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530392    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530400    2577 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530404    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530406    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530409    2577 flags.go:64] FLAG: --pod-cidr=""
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530412    2577 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530416    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530419    2577 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530421    2577 flags.go:64] FLAG: --port="10250"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530424    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530427    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c969980dfd1b6c16"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530431    2577 flags.go:64] FLAG: --qos-reserved=""
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530433    2577 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530437    2577 flags.go:64] FLAG: --register-node="true"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530439    2577 flags.go:64] FLAG: --register-with-taints=""
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530447    2577 flags.go:64] FLAG: --registry-burst="10"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530450    2577 flags.go:64] FLAG: --registry-qps="5"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530453    2577 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530456    2577 flags.go:64] FLAG: --reserved-memory=""
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530460    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530463    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530466    2577 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 07:27:45.535904 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530469    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530471    2577 flags.go:64] FLAG: --runonce="false"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530474    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530478    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530481    2577 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530484    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530487    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530492    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530496    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530499    2577 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530502    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530505    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530508    2577 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530511    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530514    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530517    2577 flags.go:64] FLAG: --system-cgroups=""
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530520    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530525    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530528    2577 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530531    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530535    2577 flags.go:64] FLAG: --tls-min-version=""
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530538    2577 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530541    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530544    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530547    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 07:27:45.536610 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530550    2577 flags.go:64] FLAG: --v="2"
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530554    2577 flags.go:64] FLAG: --version="false"
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530559    2577 flags.go:64] FLAG: --vmodule=""
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530563    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.530566    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530723    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530727    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530730    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530733    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530736    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530740    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530744    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530747    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530750    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530754    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530756    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530759    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530761    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530764    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530767    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:27:45.537622 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530769    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530772    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530774    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530777    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530779    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530782    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530785    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530788    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530790    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530793    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530795    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530798    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530801    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530803    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530806    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530808    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530810    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530813    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530816    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:27:45.538260 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530818    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530821    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530823    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530827    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530830    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530833    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530835    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530839    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530842    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530845    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530848    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530850    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530853    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530856    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530858    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530861    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530863    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530866    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530868    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530871    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:27:45.538733 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530874    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530876    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530879    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530882    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530885    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530887    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530890    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530893    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530895    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530898    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530901    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530903    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530906    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530908    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530911    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530915    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530917    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530920    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530923    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530927    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:27:45.539312 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530929    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530932    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530935    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530937    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530940    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530942    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530945    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530948    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530950    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530953    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530955    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530958    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530960    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530963    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530966    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530969    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530973    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530976    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530979    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:27:45.539802 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530982    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530984    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530987    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530989    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530992    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530994    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530997    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.530999    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531004    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531008    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531012    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531015    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531025    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.531053    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531085    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531091    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:27:45.540240 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531094    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531097    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531100    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531104    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531106    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531109    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531112    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531114    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531117    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531119    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531122    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531124    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531127    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531129    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531132    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531134    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531137    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531140    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531142    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531145    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:27:45.540659 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531147    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531150    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531152    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531155    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531157    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531161    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531164    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531166    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531170    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531174    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531176    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531179    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531181    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531185    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531188    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531192    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531195    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531199    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531202    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:27:45.541132 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531205    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531207    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531210    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531213    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531215    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531218    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531222    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531225    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531228    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531231    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531234    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531237    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531239    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531242    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531244    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531247    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531249    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531252    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531256    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531259    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:27:45.541704 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531261    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531265    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531268    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531271    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531273    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531276    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531291    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531294    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531296    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531299    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531302    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531304    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531307    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531310    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531312    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531315    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531317    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531320    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531322    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:27:45.542180 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531325    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531327    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531330    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531332    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531335    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531337    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531340    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531342    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531345    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531347    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531350    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531354    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531356    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531359    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531363    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531366    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531368    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531372    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531375    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531377    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:27:45.542639 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531380    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531382    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531385    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531388    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531390    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.531393    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.538938    2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.538956    2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.539557    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.539563    2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539627    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539632    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539636    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539640    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539643    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539647    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539649    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539652    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539655    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:27:45.543196 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539657    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539660    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539663    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539666    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539669    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539671    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539674    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539677    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539679    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539682    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539684    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539687    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539691    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539695    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539698    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539701    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539705    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539707    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539710    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:27:45.543660 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539713    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539717    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539720    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539723    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539726    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539728    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539731    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539733    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539736    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539739    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539741    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539744    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539747    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539749    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539752    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539754    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539757    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539759    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539762    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:27:45.544234 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539765    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539767    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539769    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539772    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539775    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539777    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539779    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539782    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539785    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539787    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539790    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539793    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539795    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539798    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539801    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539805    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539808    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539811    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539814    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539816    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:27:45.544695 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539819    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539822    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539825    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539827    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539830    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539832    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539835    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539837    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539840    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539842    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539845    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539847    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539851    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539854    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539857    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539859    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539862    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539864    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539867    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539870    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:27:45.545154 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539872    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539875    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539877    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539880    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539882    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539885    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539888    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539891    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539893    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539896    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539898    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539901    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539904    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539906    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539909    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539911    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539914    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539918    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:27:45.545673 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.539921    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.539950    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540093    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540097    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540100    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540103    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540105    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540109    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540112    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540114    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540117    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540119    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540122    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540125    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540127    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540130    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:27:45.546109 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540132    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540135    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540137    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540140    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540143    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540147    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540150    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540153    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540156    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540160    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540163    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540166    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540169    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540172    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540174    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540177    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540179    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540182    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540184    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:27:45.546564 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540187    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540189    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540192    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540194    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540197    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540201    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540203    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540206    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540208    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540211    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540213    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540216    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540218    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540221    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540223    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540226    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540228    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540231    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540234    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540237    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:27:45.547008 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540239    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540242    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540245    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540248    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540250    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540254    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540257    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540259    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540261    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540265    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540268    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540270    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540273    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540275    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540291    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540296    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540300    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540304    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540309    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540312    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:27:45.547577 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540315    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540318    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540320    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540323    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540326    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540329    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540331    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540334    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540336    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540339    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540341    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540344    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540347    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540349    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540352    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540355    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540357    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540360    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540362    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540365    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:27:45.548377 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540367    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540370    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540372    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540377    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540381    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540384    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540387    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540389    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540392    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540395    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540397    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540400    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540403    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.540427    2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540461    2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540465    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:27:45.548997 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540468    2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540472    2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540475    2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540478    2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540481    2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540483    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540486    2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540489    2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540491    2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540494    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540497    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540499    2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540502    2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540505    2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540508    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540510    2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540512    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540515    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540518    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540521    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:27:45.549469 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540523    2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540527    2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540531    2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540534    2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540537    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540539    2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540542    2577 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540544    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540548    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540550    2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540553    2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540555    2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540558    2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540560    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540563    2577 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540565    2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540568    2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540571    2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540573    2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:27:45.549990 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540575    2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540578    2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540580    2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540583    2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540585    2577 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540588    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540591    2577 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540593    2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540595    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540598    2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540600    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540603    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540605    2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540608    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540611    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540613    2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540616    2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540618    2577 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540621    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:27:45.550563 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540624    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540626    2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540629    2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540637    2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540640    2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540643    2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540646    2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540648    2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540651    2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540654    2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540656    2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540659    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540661    2577 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540664    2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540666    2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540669    2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540672    2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540674    2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540677    2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540679    2577 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:27:45.551145 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540682    2577 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540684    2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540687    2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540689    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540692    2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540694    2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540697    2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540700    2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540703    2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540705    2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540708    2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540710    2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540713    2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540719    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540721    2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540724    2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540729    2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540733    2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540736    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:27:45.551646 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540739    2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540742    2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540745    2577 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540749    2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540751    2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540754    2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:45.540756    2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.540824    2577 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.542690    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.543868    2577 server.go:1014] "Starting client certificate rotation"
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.543989    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.544039    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.546584    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:27:45.552127 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.549468    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:27:45.562107 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.562087    2577 log.go:25] "Validated CRI v1 runtime API"
Oct 31 07:27:45.565957 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.565938    2577 log.go:25] "Validated CRI v1 image API"
Oct 31 07:27:45.566854 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.566826    2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 07:27:45.570225 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.570199    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d173a70e-bbe9-4b8c-bf85-09694211657f:/dev/nvme0n1p4 f82f9d83-cdba-4c31-b103-4aceb0aaae16:/dev/nvme0n1p3]
Oct 31 07:27:45.570326 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.570222    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 07:27:45.575243 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.575118    2577 manager.go:217] Machine: {Timestamp:2026-10-31 07:27:45.574619778 +0000 UTC m=+0.372303877 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099964 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ebf9231d6dbdcc605db03d1dffdd3 SystemUUID:ec2ebf92-31d6-dbdc-c605-db03d1dffdd3 BootID:dc38b392-66b2-4898-8795-69410b88ea50 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:1f:1f:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:1f:1f:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:a2:f0:97:a1:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 07:27:45.575243 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.575228    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 07:27:45.575422 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.575355    2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 07:27:45.576481 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.576453    2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 07:27:45.576673 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.576483    2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 07:27:45.576721 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.576681    2577 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 07:27:45.576721 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.576692    2577 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 07:27:45.576770 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.576741    2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:27:45.576770 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.576764    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:27:45.578905 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.578885    2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 07:27:45.582701 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.582685    2577 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 07:27:45.582840 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.582828    2577 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 07:27:45.583913 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.583901    2577 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 07:27:45.583969 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.583920    2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 07:27:45.583969 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.583935    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 07:27:45.583969 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.583949    2577 kubelet.go:405] "Adding apiserver pod source"
Oct 31 07:27:45.583969 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.583961    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 07:27:45.584401 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.584379    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:27:45.587086 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.587067    2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 07:27:45.591945 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.591933    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:27:45.592638 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.592625    2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 07:27:45.592672 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.592657    2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 07:27:45.592890 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.592874    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 07:27:45.592931 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.592904    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 07:27:45.592931 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.592913    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 07:27:45.592931 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.592919    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 07:27:45.592931 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.592925    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 07:27:45.592931 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.592930    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 07:27:45.593291 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.593271    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 07:27:45.593323 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.593297    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 07:27:45.593323 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.593307    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 07:27:45.593323 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.593313    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 07:27:45.593323 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.593322    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 07:27:45.593481 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.593330    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 07:27:45.593943 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.593934    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 07:27:45.593992 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.593944    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 07:27:45.598214 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.598199    2577 server.go:1263] "Started kubelet"
Oct 31 07:27:45.598320 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.598261    2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 07:27:45.599000 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.598855    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 07:27:45.599068 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.599056    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 07:27:45.599173 ip-10-0-140-236 systemd[1]: Started Kubernetes Kubelet.
Oct 31 07:27:45.601013 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.600987    2577 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 07:27:45.602088 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.602070    2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 07:27:45.603973 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.603951    2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 07:27:45.606650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.606628    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 07:27:45.607296 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.607266    2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 07:27:45.607413 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.607371    2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found"
Oct 31 07:27:45.607979 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.607959    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 07:27:45.607979 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.607971    2577 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 07:27:45.608118 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.607987    2577 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 07:27:45.608378 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.608363    2577 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 07:27:45.608378 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.608376    2577 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 07:27:45.608481 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.608423    2577 factory.go:55] Registering systemd factory
Oct 31 07:27:45.608481 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.608435    2577 factory.go:223] Registration of the systemd container factory successfully
Oct 31 07:27:45.608694 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.608679    2577 factory.go:153] Registering CRI-O factory
Oct 31 07:27:45.608694 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.608691    2577 factory.go:223] Registration of the crio container factory successfully
Oct 31 07:27:45.608802 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.608739    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 07:27:45.608802 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.608755    2577 factory.go:103] Registering Raw factory
Oct 31 07:27:45.608802 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.608765    2577 manager.go:1199] Started watching for new ooms in manager
Oct 31 07:27:45.609673 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.609653    2577 manager.go:319] Starting recovery of all containers
Oct 31 07:27:45.622529 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.622504    2577 manager.go:324] Recovery completed
Oct 31 07:27:45.627198 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.627181    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:27:45.629077 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.629052    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 07:27:45.629160 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.629096    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 07:27:45.629160 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.629112    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID"
Oct 31 07:27:45.629705 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.629689    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:27:45.629772 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.629693    2577 cpu_manager.go:226] "Starting" policy="none"
Oct 31 07:27:45.629772 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.629726    2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 07:27:45.629772 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.629748    2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 07:27:45.631096 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.631076    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 07:27:45.631180 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.631111    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 07:27:45.631180 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.631125    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID"
Oct 31 07:27:45.632031 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.632019    2577 policy_none.go:50] "Start"
Oct 31 07:27:45.632094 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.632035    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 07:27:45.632094 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.632045    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 07:27:45.634029 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.634019    2577 policy_none.go:44] "Start"
Oct 31 07:27:45.635408 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.635379    2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 07:27:45.635473 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.635375    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 07:27:45.636294 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.635376    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-236.ec2.internal.18e38c990ce7f0cc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-236.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-236.ec2.internal,},FirstTimestamp:2026-10-31 07:27:45.598001356 +0000 UTC m=+0.395685454,LastTimestamp:2026-10-31 07:27:45.598001356 +0000 UTC m=+0.395685454,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-236.ec2.internal,}"
Oct 31 07:27:45.636470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.636453    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 07:27:45.636532 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.636493    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s26qm"
Oct 31 07:27:45.655515 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.655379    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-236.ec2.internal.18e38c990ec22158  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-236.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-236.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-236.ec2.internal,},FirstTimestamp:2026-10-31 07:27:45.629077848 +0000 UTC m=+0.426761948,LastTimestamp:2026-10-31 07:27:45.629077848 +0000 UTC m=+0.426761948,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-236.ec2.internal,}"
Oct 31 07:27:45.655809 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.655789    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 07:27:45.668527 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.668507    2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 07:27:45.675908 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.675870    2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 07:27:45.678343 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.678315    2577 manager.go:341] "Starting Device Plugin manager"
Oct 31 07:27:45.678431 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.678353    2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 07:27:45.678431 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.678366    2577 server.go:87] "Starting device plugin registration server"
Oct 31 07:27:45.678684 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.678658    2577 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 07:27:45.678684 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.678671    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 07:27:45.678870 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.678850    2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 07:27:45.678947 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.678936    2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 07:27:45.678947 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.678944    2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 07:27:45.679569 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.679390    2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 07:27:45.679653 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.679596    2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-236.ec2.internal\" not found"
Oct 31 07:27:45.680299 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.680265    2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:27:45.736392 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.736353    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 07:27:45.738439 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.737590    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 07:27:45.738439 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.737613    2577 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 07:27:45.738439 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.737633    2577 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 07:27:45.738439 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.737672    2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 07:27:45.751754 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.751738    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:27:45.779015 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.778930    2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:27:45.779851 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.779836    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 07:27:45.779923 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.779867    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 07:27:45.779923 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.779879    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID"
Oct 31 07:27:45.779923 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.779916    2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-236.ec2.internal"
Oct 31 07:27:45.797848 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.797829    2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-236.ec2.internal"
Oct 31 07:27:45.797923 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:45.797852    2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-236.ec2.internal\": node \"ip-10-0-140-236.ec2.internal\" not found"
Oct 31 07:27:45.838008 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.837977    2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"]
Oct 31 07:27:45.840407 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.840391    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:45.842288 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.842265    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:45.870766 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.870750    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:45.875352 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.875319    2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:45.903811 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.903779    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:27:45.903960 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.903828    2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:27:45.909312 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.909276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:45.909400 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.909318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:45.909400 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:45.909336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:46.009478 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.009440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:46.009478 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.009473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:46.009684 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.009490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:46.009684 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.009545    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:46.009684 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.009549    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:46.009684 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.009595    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:46.211790 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.211742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:46.211973 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.211817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:27:46.543392 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.543358    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 07:27:46.543901 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.543549    2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 07:27:46.585158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.585133    2577 apiserver.go:52] "Watching apiserver"
Oct 31 07:27:46.607700 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.607672    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 07:27:46.624577 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.624545    2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:27:46.640183 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.640146    2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 07:27:46.640508 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.640488    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-wbg8z","openshift-network-operator/iptables-alerter-96gd4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal","openshift-multus/multus-5jj86","kube-system/konnectivity-agent-pkj5d","openshift-dns/node-resolver-v59hm","openshift-multus/multus-additional-cni-plugins-cc6gh","openshift-multus/network-metrics-daemon-s2r9l","kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal","openshift-network-diagnostics/network-check-target-zb9td","openshift-ovn-kubernetes/ovnkube-node-4g4jz","kube-system/global-pull-secret-syncer-82457","openshift-cluster-node-tuning-operator/tuned-86nzk"]
Oct 31 07:27:46.642203 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.642177    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:22:45 +0000 UTC" deadline="2028-07-31 07:09:00.689128048 +0000 UTC"
Oct 31 07:27:46.642249 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.642204    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15335h41m14.046926934s"
Oct 31 07:27:46.643186 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.643174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.645474 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.645453    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:46.645580 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.645525    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:27:46.647730 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.647715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.650025 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.650007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.652405 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.652383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:46.654721 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.654704    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:46.654826 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.654773    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:27:46.656931 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.656915    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.659194 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.659179    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.660810 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.660790    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 07:27:46.660903 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.660878    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xtmc6\""
Oct 31 07:27:46.660948 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.660935    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 07:27:46.661501 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.661478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.664138 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.664117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.666197 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.666175    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7m4f\""
Oct 31 07:27:46.668086 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.667503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.670741 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.670723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:46.670817 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.670781    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:27:46.671781 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.671763    2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 07:27:46.671840 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.671817    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fs2g9"
Oct 31 07:27:46.709060 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.709043    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 07:27:46.714272 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.714397 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66fgg\" (UniqueName: \"kubernetes.io/projected/836cf52b-b0e8-4e98-a812-21d0cebfaf63-kube-api-access-66fgg\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.714397 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714309    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-hosts-file\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.714397 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-lib-modules\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.714397 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714342    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-tmp\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.714397 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-systemd-units\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.714397 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714379    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-var-lib-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.714599 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-netd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.714599 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.714599 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc187c9-2163-4de8-afbb-77f65de777c8-ovn-node-metrics-cert\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.714599 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714509    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbx2j\" (UniqueName: \"kubernetes.io/projected/f206a32f-819d-4e61-9040-790e8c9183ec-kube-api-access-xbx2j\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.714599 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:46.714599 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-host\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.714599 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f206a32f-819d-4e61-9040-790e8c9183ec-serviceca\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.714808 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-netns\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.714808 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-dbus\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:46.714808 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714681    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.714808 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714704    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-run\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.714808 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714756    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f206a32f-819d-4e61-9040-790e8c9183ec-host\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.714808 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714798    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r8cv\" (UniqueName: \"kubernetes.io/projected/4fc187c9-2163-4de8-afbb-77f65de777c8-kube-api-access-4r8cv\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.714983 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.714983 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-daemon-config\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.714983 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-kubernetes\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.714983 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvhrd\" (UniqueName: \"kubernetes.io/projected/2af799e3-eb79-41f6-be34-411ea2f10326-kube-api-access-cvhrd\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.715102 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-ovn\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715102 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.714999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-env-overrides\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715102 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsnkc\" (UniqueName: \"kubernetes.io/projected/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-kube-api-access-nsnkc\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.715102 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715068    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-registration-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.715102 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/132e4deb-32cc-4b7b-a179-44a9700f1bac-konnectivity-ca\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:46.715276 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-kubelet\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715276 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-script-lib\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715276 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.715276 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715276 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-cnibin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715276 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715224    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-hostroot\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715276 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/132e4deb-32cc-4b7b-a179-44a9700f1bac-agent-certs\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:46.715276 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-etc-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715301    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-system-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-multus\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-conf-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2af799e3-eb79-41f6-be34-411ea2f10326-iptables-alerter-script\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-bin\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-conf\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715495    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-log-socket\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-socket-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7vps\" (UniqueName: \"kubernetes.io/projected/db612775-57b2-405c-a403-8c78f173d018-kube-api-access-g7vps\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.715585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-k8s-cni-cncf-io\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715597    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-kubelet\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-etc-kubernetes\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715654    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-multus-certs\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715784    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-systemd\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-tuned\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-slash\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.715873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-netns\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-socket-dir-parent\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715922    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-bin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715945    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-modprobe-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-cni-binary-copy\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.715987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716038    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-kubelet-config\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-var-lib-kubelet\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-sys\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.716193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-systemd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716208    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-config\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx78k\" (UniqueName: \"kubernetes.io/projected/1e3a7ead-4527-476b-9b02-c50e68e80166-kube-api-access-cx78k\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkpsk\" (UniqueName: \"kubernetes.io/projected/cee91069-0b51-442c-a3c3-71e71302b0e5-kube-api-access-lkpsk\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7b5w\" (UniqueName: \"kubernetes.io/projected/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-kube-api-access-m7b5w\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716329    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysconfig\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2af799e3-eb79-41f6-be34-411ea2f10326-host-slash\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-node-log\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-tmp-dir\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-sys-fs\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716482    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cnibin\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.716552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.716505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-device-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.770186 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770155    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 07:27:46.770433 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770406    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x6ssn\""
Oct 31 07:27:46.770565 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770465    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xhs6v\""
Oct 31 07:27:46.770565 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770522    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgg5g\""
Oct 31 07:27:46.770565 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770532    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:27:46.770743 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770571    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 07:27:46.770743 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770626    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 07:27:46.770743 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770636    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgtls\""
Oct 31 07:27:46.770743 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770659    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 07:27:46.770960 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770831    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 07:27:46.770960 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770870    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 07:27:46.770960 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770888    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 07:27:46.771118 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770966    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 07:27:46.771118 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.770987    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 07:27:46.771222 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771119    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 07:27:46.771222 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771129    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh9bq\""
Oct 31 07:27:46.771222 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771124    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 07:27:46.771222 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771162    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7l5b\""
Oct 31 07:27:46.771222 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771179    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771229    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771251    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771278    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771130    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gjb5j\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771308    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771336    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771377    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771393    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771394    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771407    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771417    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771447    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:27:46.771494 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.771474    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 07:27:46.817153 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-lib-modules\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.817153 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-tmp\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.817388 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-systemd-units\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.817388 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817196    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-var-lib-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.817388 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817255    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-var-lib-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.817388 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817259    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-systemd-units\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.817388 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-netd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.817388 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.817388 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817368    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-lib-modules\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc187c9-2163-4de8-afbb-77f65de777c8-ovn-node-metrics-cert\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817408    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817411    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-netd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbx2j\" (UniqueName: \"kubernetes.io/projected/f206a32f-819d-4e61-9040-790e8c9183ec-kube-api-access-xbx2j\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-host\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f206a32f-819d-4e61-9040-790e8c9183ec-serviceca\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817598    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-host\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817591    2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-netns\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-dbus\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:46.817681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817678    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-run\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817706    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-netns\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f206a32f-819d-4e61-9040-790e8c9183ec-host\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8cv\" (UniqueName: \"kubernetes.io/projected/4fc187c9-2163-4de8-afbb-77f65de777c8-kube-api-access-4r8cv\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817770    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817807    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-daemon-config\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817853    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-run\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-kubernetes\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817835    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817881    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f206a32f-819d-4e61-9040-790e8c9183ec-host\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.818158 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.817861    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-dbus\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818240    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhrd\" (UniqueName: \"kubernetes.io/projected/2af799e3-eb79-41f6-be34-411ea2f10326-kube-api-access-cvhrd\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818306    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-ovn\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-env-overrides\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnkc\" (UniqueName: \"kubernetes.io/projected/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-kube-api-access-nsnkc\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-registration-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/132e4deb-32cc-4b7b-a179-44a9700f1bac-konnectivity-ca\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818489    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f206a32f-819d-4e61-9040-790e8c9183ec-serviceca\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-kubelet\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818529    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-kubernetes\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-script-lib\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818600    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-ovn\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.818650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-cnibin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818665    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818689    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-hostroot\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818712    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-kubelet\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/132e4deb-32cc-4b7b-a179-44a9700f1bac-agent-certs\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-etc-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-system-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818827    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-multus\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818858    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-conf-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2af799e3-eb79-41f6-be34-411ea2f10326-iptables-alerter-script\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.818969    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-bin\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.819282 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819079    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/132e4deb-32cc-4b7b-a179-44a9700f1bac-konnectivity-ca\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:46.819853 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819350    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-script-lib\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.819853 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819448    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-env-overrides\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.819853 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819452    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.819853 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819528    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-etc-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.819853 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819592    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-system-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.819853 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819644    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-multus\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.819853 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819695    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-conf-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.820213 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819936    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-cnibin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.820213 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.819986    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-hostroot\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.820348 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820210    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-daemon-config\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.820348 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820313    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-registration-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.820629 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-bin\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.820749 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-conf\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.820749 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820690    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-log-socket\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.820749 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-socket-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.820890 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vps\" (UniqueName: \"kubernetes.io/projected/db612775-57b2-405c-a403-8c78f173d018-kube-api-access-g7vps\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.820890 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820797    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-k8s-cni-cncf-io\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.820890 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820825    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-kubelet\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.820890 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-etc-kubernetes\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.821075 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-multus-certs\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.821075 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.821075 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.821075 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.820992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.821075 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.821075 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-systemd\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.821341 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-tuned\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.821341 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-slash\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.821341 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-netns\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.821341 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-socket-dir-parent\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.821341 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821230    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-bin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.821341 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821261    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-modprobe-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.821341 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-cni-binary-copy\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.821341 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-kubelet-config\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-var-lib-kubelet\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-sys\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-systemd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-config\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821638    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx78k\" (UniqueName: \"kubernetes.io/projected/1e3a7ead-4527-476b-9b02-c50e68e80166-kube-api-access-cx78k\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkpsk\" (UniqueName: \"kubernetes.io/projected/cee91069-0b51-442c-a3c3-71e71302b0e5-kube-api-access-lkpsk\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7b5w\" (UniqueName: \"kubernetes.io/projected/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-kube-api-access-m7b5w\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.821728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821730    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysconfig\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821760    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2af799e3-eb79-41f6-be34-411ea2f10326-host-slash\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-node-log\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-tmp-dir\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821837    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2af799e3-eb79-41f6-be34-411ea2f10326-iptables-alerter-script\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821845    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-sys-fs\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cnibin\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-device-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821943    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821972    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-modprobe-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.821975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fgg\" (UniqueName: \"kubernetes.io/projected/836cf52b-b0e8-4e98-a812-21d0cebfaf63-kube-api-access-66fgg\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-hosts-file\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822083    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-tmp\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822154    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-hosts-file\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.822274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822267    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-conf\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822313    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc187c9-2163-4de8-afbb-77f65de777c8-ovn-node-metrics-cert\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822324    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-log-socket\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822465    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-var-lib-kubelet\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822579    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-socket-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822752    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysconfig\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822818    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2af799e3-eb79-41f6-be34-411ea2f10326-host-slash\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822870    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-node-log\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822896    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-cni-binary-copy\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822900    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-k8s-cni-cncf-io\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.822999 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.822957    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-kubelet\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823015    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-etc-kubernetes\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823074    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-multus-certs\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823077    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823133    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823152    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-sys\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-tmp-dir\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823219    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823221    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823271    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-sys-fs\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823294    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-kubelet-config\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823282    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-systemd\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823375    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cnibin\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823394    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-socket-dir-parent\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.823444    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:46.823470 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823462    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-bin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823513    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.823521    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret podName:f0f4a456-26c2-4a45-bd30-b6795d5c03cc nodeName:}" failed. No retries permitted until 2026-10-31 07:27:47.323499524 +0000 UTC m=+2.121183635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret") pod "global-pull-secret-syncer-82457" (UID: "f0f4a456-26c2-4a45-bd30-b6795d5c03cc") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.823600    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.823666    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs podName:cee91069-0b51-442c-a3c3-71e71302b0e5 nodeName:}" failed. No retries permitted until 2026-10-31 07:27:47.323641753 +0000 UTC m=+2.121325843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs") pod "network-metrics-daemon-s2r9l" (UID: "cee91069-0b51-442c-a3c3-71e71302b0e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823662    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-config\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-systemd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823745    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-device-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823818    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-netns\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.823888    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-slash\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.824203 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.824186    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.824849 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.824828    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.825578 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.825556    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-tuned\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.827139 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.827114    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/132e4deb-32cc-4b7b-a179-44a9700f1bac-agent-certs\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:46.836264 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.836241    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:27:46.836264 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.836265    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:27:46.836430 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.836280    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4944 for pod openshift-network-diagnostics/network-check-target-zb9td: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:46.836430 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:46.836346    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944 podName:8856b22a-b791-4ca1-8083-650b391429af nodeName:}" failed. No retries permitted until 2026-10-31 07:27:47.336327288 +0000 UTC m=+2.134011403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h4944" (UniqueName: "kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944") pod "network-check-target-zb9td" (UID: "8856b22a-b791-4ca1-8083-650b391429af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:46.838322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.838299    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbx2j\" (UniqueName: \"kubernetes.io/projected/f206a32f-819d-4e61-9040-790e8c9183ec-kube-api-access-xbx2j\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:46.840171 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.840141    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnkc\" (UniqueName: \"kubernetes.io/projected/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-kube-api-access-nsnkc\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.840757 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.840732    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhrd\" (UniqueName: \"kubernetes.io/projected/2af799e3-eb79-41f6-be34-411ea2f10326-kube-api-access-cvhrd\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:46.841083 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.841064    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8cv\" (UniqueName: \"kubernetes.io/projected/4fc187c9-2163-4de8-afbb-77f65de777c8-kube-api-access-4r8cv\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:46.842963 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.842938    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkpsk\" (UniqueName: \"kubernetes.io/projected/cee91069-0b51-442c-a3c3-71e71302b0e5-kube-api-access-lkpsk\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:46.843046 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.842986    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fgg\" (UniqueName: \"kubernetes.io/projected/836cf52b-b0e8-4e98-a812-21d0cebfaf63-kube-api-access-66fgg\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:46.843716 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.843698    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vps\" (UniqueName: \"kubernetes.io/projected/db612775-57b2-405c-a403-8c78f173d018-kube-api-access-g7vps\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:46.843855 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.843832    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx78k\" (UniqueName: \"kubernetes.io/projected/1e3a7ead-4527-476b-9b02-c50e68e80166-kube-api-access-cx78k\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:27:46.844297 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.844274    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7b5w\" (UniqueName: \"kubernetes.io/projected/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-kube-api-access-m7b5w\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:46.895691 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:46.895657    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bef5910d750df47fec5174fe909f02.slice/crio-cce489912507a7128eabc59d8e95f8d1145bf3fb49f84ca8f2639aefba65e84b WatchSource:0}: Error finding container cce489912507a7128eabc59d8e95f8d1145bf3fb49f84ca8f2639aefba65e84b: Status 404 returned error can't find the container with id cce489912507a7128eabc59d8e95f8d1145bf3fb49f84ca8f2639aefba65e84b
Oct 31 07:27:46.896130 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:46.896111    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82d0dbde7394ec25631244e1f1d5cd9.slice/crio-8a1106ff94c3a8aa7b24c8e86d66006d4fa4e9d6ac46c1f682b5eee6af2ed14e WatchSource:0}: Error finding container 8a1106ff94c3a8aa7b24c8e86d66006d4fa4e9d6ac46c1f682b5eee6af2ed14e: Status 404 returned error can't find the container with id 8a1106ff94c3a8aa7b24c8e86d66006d4fa4e9d6ac46c1f682b5eee6af2ed14e
Oct 31 07:27:46.901207 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.901189    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:27:46.974182 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.974138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:27:46.985471 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:46.985446    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab61d51b_bab9_4ed6_aaa0_1d5e054fefb7.slice/crio-826e797743f1a8ab0415c3d272c61f0319adfa5509a314b51673345fe47f394d WatchSource:0}: Error finding container 826e797743f1a8ab0415c3d272c61f0319adfa5509a314b51673345fe47f394d: Status 404 returned error can't find the container with id 826e797743f1a8ab0415c3d272c61f0319adfa5509a314b51673345fe47f394d
Oct 31 07:27:46.996322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:46.996300    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:27:47.005172 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:47.005146    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb612775_57b2_405c_a403_8c78f173d018.slice/crio-1c50ba8c8f775968ea83b50a1cffe3e94b895e7dd2c0f6d7da6fc47edcb573d2 WatchSource:0}: Error finding container 1c50ba8c8f775968ea83b50a1cffe3e94b895e7dd2c0f6d7da6fc47edcb573d2: Status 404 returned error can't find the container with id 1c50ba8c8f775968ea83b50a1cffe3e94b895e7dd2c0f6d7da6fc47edcb573d2
Oct 31 07:27:47.005258 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.005200    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5jj86"
Oct 31 07:27:47.014252 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:47.014229    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3a7ead_4527_476b_9b02_c50e68e80166.slice/crio-4135f17c15b08ddff40838d0874e901eb5560c4d09c07d2e62eb661c88c198cb WatchSource:0}: Error finding container 4135f17c15b08ddff40838d0874e901eb5560c4d09c07d2e62eb661c88c198cb: Status 404 returned error can't find the container with id 4135f17c15b08ddff40838d0874e901eb5560c4d09c07d2e62eb661c88c198cb
Oct 31 07:27:47.023903 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.023882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:47.033638 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:47.033614    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod132e4deb_32cc_4b7b_a179_44a9700f1bac.slice/crio-e23e9f7f6762228112a7c326718a42286611483fda8838fa797b01cbf9c19090 WatchSource:0}: Error finding container e23e9f7f6762228112a7c326718a42286611483fda8838fa797b01cbf9c19090: Status 404 returned error can't find the container with id e23e9f7f6762228112a7c326718a42286611483fda8838fa797b01cbf9c19090
Oct 31 07:27:47.039902 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.039880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:27:47.048768 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:47.048748    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836cf52b_b0e8_4e98_a812_21d0cebfaf63.slice/crio-3509002e707ffa1a0ca9085c67e967a5b7c7efd6ef8b88f1be8a4704d00a3cd2 WatchSource:0}: Error finding container 3509002e707ffa1a0ca9085c67e967a5b7c7efd6ef8b88f1be8a4704d00a3cd2: Status 404 returned error can't find the container with id 3509002e707ffa1a0ca9085c67e967a5b7c7efd6ef8b88f1be8a4704d00a3cd2
Oct 31 07:27:47.058974 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.058955    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:27:47.068437 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:47.068416    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2af799e3_eb79_41f6_be34_411ea2f10326.slice/crio-b10279343c85685543df2671b87124ea3283151b3c3ce9771b68f6fc1bbdba49 WatchSource:0}: Error finding container b10279343c85685543df2671b87124ea3283151b3c3ce9771b68f6fc1bbdba49: Status 404 returned error can't find the container with id b10279343c85685543df2671b87124ea3283151b3c3ce9771b68f6fc1bbdba49
Oct 31 07:27:47.079367 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.079341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:27:47.089262 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:47.089239    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf206a32f_819d_4e61_9040_790e8c9183ec.slice/crio-dcc41609851bea2369d498de074702337e8bdda82cb0dff5822362ffc4d85974 WatchSource:0}: Error finding container dcc41609851bea2369d498de074702337e8bdda82cb0dff5822362ffc4d85974: Status 404 returned error can't find the container with id dcc41609851bea2369d498de074702337e8bdda82cb0dff5822362ffc4d85974
Oct 31 07:27:47.117379 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.117351    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:27:47.128200 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:47.128174    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc187c9_2163_4de8_afbb_77f65de777c8.slice/crio-2b922b9e6f40c1835b16f980b82d23e256f7f2527e59a0553cd0804c32f83ea8 WatchSource:0}: Error finding container 2b922b9e6f40c1835b16f980b82d23e256f7f2527e59a0553cd0804c32f83ea8: Status 404 returned error can't find the container with id 2b922b9e6f40c1835b16f980b82d23e256f7f2527e59a0553cd0804c32f83ea8
Oct 31 07:27:47.128961 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.128943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:27:47.136796 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:27:47.136776    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486ea8a8_d6fe_4a72_b04f_dbd51bc39fdb.slice/crio-b72f45ccc31af5137314f6e552b5db1ff3eae84c2e41fcc5cd2b38593ebb6cee WatchSource:0}: Error finding container b72f45ccc31af5137314f6e552b5db1ff3eae84c2e41fcc5cd2b38593ebb6cee: Status 404 returned error can't find the container with id b72f45ccc31af5137314f6e552b5db1ff3eae84c2e41fcc5cd2b38593ebb6cee
Oct 31 07:27:47.156183 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.156160    2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:27:47.326750 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.326664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:47.326750 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.326705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:47.326956 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:47.326823    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:47.326956 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:47.326888    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret podName:f0f4a456-26c2-4a45-bd30-b6795d5c03cc nodeName:}" failed. No retries permitted until 2026-10-31 07:27:48.326869684 +0000 UTC m=+3.124553787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret") pod "global-pull-secret-syncer-82457" (UID: "f0f4a456-26c2-4a45-bd30-b6795d5c03cc") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:47.326956 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:47.326823    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:47.327113 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:47.326997    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs podName:cee91069-0b51-442c-a3c3-71e71302b0e5 nodeName:}" failed. No retries permitted until 2026-10-31 07:27:48.326974342 +0000 UTC m=+3.124658452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs") pod "network-metrics-daemon-s2r9l" (UID: "cee91069-0b51-442c-a3c3-71e71302b0e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:47.428053 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.428012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:47.428224 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:47.428177    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:27:47.428224 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:47.428197    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:27:47.428224 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:47.428210    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4944 for pod openshift-network-diagnostics/network-check-target-zb9td: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:47.428401 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:47.428269    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944 podName:8856b22a-b791-4ca1-8083-650b391429af nodeName:}" failed. No retries permitted until 2026-10-31 07:27:48.428250923 +0000 UTC m=+3.225935030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4944" (UniqueName: "kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944") pod "network-check-target-zb9td" (UID: "8856b22a-b791-4ca1-8083-650b391429af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:47.675549 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.675436    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:22:46 +0000 UTC" deadline="2028-05-14 13:25:17.634533259 +0000 UTC"
Oct 31 07:27:47.675549 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.675499    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13469h57m29.959037758s"
Oct 31 07:27:47.740918 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.740882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:47.741082 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:47.741025    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:27:47.760221 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.760157    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wbg8z" event={"ID":"f206a32f-819d-4e61-9040-790e8c9183ec","Type":"ContainerStarted","Data":"dcc41609851bea2369d498de074702337e8bdda82cb0dff5822362ffc4d85974"}
Oct 31 07:27:47.770997 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.770962    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-96gd4" event={"ID":"2af799e3-eb79-41f6-be34-411ea2f10326","Type":"ContainerStarted","Data":"b10279343c85685543df2671b87124ea3283151b3c3ce9771b68f6fc1bbdba49"}
Oct 31 07:27:47.786574 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.786536    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl" event={"ID":"db612775-57b2-405c-a403-8c78f173d018","Type":"ContainerStarted","Data":"1c50ba8c8f775968ea83b50a1cffe3e94b895e7dd2c0f6d7da6fc47edcb573d2"}
Oct 31 07:27:47.804745 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.804700    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v59hm" event={"ID":"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7","Type":"ContainerStarted","Data":"826e797743f1a8ab0415c3d272c61f0319adfa5509a314b51673345fe47f394d"}
Oct 31 07:27:47.822488 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.822449    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" event={"ID":"e82d0dbde7394ec25631244e1f1d5cd9","Type":"ContainerStarted","Data":"8a1106ff94c3a8aa7b24c8e86d66006d4fa4e9d6ac46c1f682b5eee6af2ed14e"}
Oct 31 07:27:47.834068 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.833971    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerStarted","Data":"cce489912507a7128eabc59d8e95f8d1145bf3fb49f84ca8f2639aefba65e84b"}
Oct 31 07:27:47.841591 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.841431    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerStarted","Data":"2b922b9e6f40c1835b16f980b82d23e256f7f2527e59a0553cd0804c32f83ea8"}
Oct 31 07:27:47.848256 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.848184    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-86nzk" event={"ID":"836cf52b-b0e8-4e98-a812-21d0cebfaf63","Type":"ContainerStarted","Data":"3509002e707ffa1a0ca9085c67e967a5b7c7efd6ef8b88f1be8a4704d00a3cd2"}
Oct 31 07:27:47.851203 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.851146    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pkj5d" event={"ID":"132e4deb-32cc-4b7b-a179-44a9700f1bac","Type":"ContainerStarted","Data":"e23e9f7f6762228112a7c326718a42286611483fda8838fa797b01cbf9c19090"}
Oct 31 07:27:47.859753 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.859688    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5jj86" event={"ID":"1e3a7ead-4527-476b-9b02-c50e68e80166","Type":"ContainerStarted","Data":"4135f17c15b08ddff40838d0874e901eb5560c4d09c07d2e62eb661c88c198cb"}
Oct 31 07:27:47.861201 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.861147    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cc6gh" event={"ID":"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb","Type":"ContainerStarted","Data":"b72f45ccc31af5137314f6e552b5db1ff3eae84c2e41fcc5cd2b38593ebb6cee"}
Oct 31 07:27:47.868870 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:47.868795    2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:27:48.336209 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:48.336169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:48.336412 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:48.336224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:48.336412 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.336374    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:48.336412 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.336385    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:48.336574 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.336434    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs podName:cee91069-0b51-442c-a3c3-71e71302b0e5 nodeName:}" failed. No retries permitted until 2026-10-31 07:27:50.336416194 +0000 UTC m=+5.134100295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs") pod "network-metrics-daemon-s2r9l" (UID: "cee91069-0b51-442c-a3c3-71e71302b0e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:48.336574 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.336451    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret podName:f0f4a456-26c2-4a45-bd30-b6795d5c03cc nodeName:}" failed. No retries permitted until 2026-10-31 07:27:50.336443307 +0000 UTC m=+5.134127398 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret") pod "global-pull-secret-syncer-82457" (UID: "f0f4a456-26c2-4a45-bd30-b6795d5c03cc") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:48.437010 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:48.436972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:48.437192 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.437172    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:27:48.437192 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.437192    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:27:48.437316 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.437205    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4944 for pod openshift-network-diagnostics/network-check-target-zb9td: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:48.437316 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.437265    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944 podName:8856b22a-b791-4ca1-8083-650b391429af nodeName:}" failed. No retries permitted until 2026-10-31 07:27:50.437246126 +0000 UTC m=+5.234930227 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4944" (UniqueName: "kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944") pod "network-check-target-zb9td" (UID: "8856b22a-b791-4ca1-8083-650b391429af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:48.676319 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:48.676198    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:22:46 +0000 UTC" deadline="2028-07-01 09:35:45.991500777 +0000 UTC"
Oct 31 07:27:48.676319 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:48.676242    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14618h7m57.315262991s"
Oct 31 07:27:48.738650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:48.738613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:48.738881 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.738854    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:27:48.739263 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:48.739247    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:48.739907 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:48.739355    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:27:49.742629 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:49.742598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:49.743205 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:49.742745    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:27:50.354153 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:50.354111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:50.354386 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:50.354170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:50.354386 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.354371    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:50.354498 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.354438    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret podName:f0f4a456-26c2-4a45-bd30-b6795d5c03cc nodeName:}" failed. No retries permitted until 2026-10-31 07:27:54.354418695 +0000 UTC m=+9.152102801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret") pod "global-pull-secret-syncer-82457" (UID: "f0f4a456-26c2-4a45-bd30-b6795d5c03cc") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:50.358297 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.354832    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:50.358297 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.354881    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs podName:cee91069-0b51-442c-a3c3-71e71302b0e5 nodeName:}" failed. No retries permitted until 2026-10-31 07:27:54.354866998 +0000 UTC m=+9.152551100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs") pod "network-metrics-daemon-s2r9l" (UID: "cee91069-0b51-442c-a3c3-71e71302b0e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:50.455505 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:50.454836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:50.455505 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.455004    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:27:50.455505 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.455028    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:27:50.455505 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.455041    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4944 for pod openshift-network-diagnostics/network-check-target-zb9td: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:50.455505 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.455101    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944 podName:8856b22a-b791-4ca1-8083-650b391429af nodeName:}" failed. No retries permitted until 2026-10-31 07:27:54.455082743 +0000 UTC m=+9.252766845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4944" (UniqueName: "kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944") pod "network-check-target-zb9td" (UID: "8856b22a-b791-4ca1-8083-650b391429af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:50.738491 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:50.738415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:50.738654 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.738569    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:27:50.738738 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:50.738715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:50.738827 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:50.738807    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:27:51.739777 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:51.739423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:51.739777 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:51.739734    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:27:51.879109 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:51.879071    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" event={"ID":"e82d0dbde7394ec25631244e1f1d5cd9","Type":"ContainerStarted","Data":"61602b9beadcf2a2d4f684dd3032c13a671318b6ea65036cc5c326dbb4d7c562"}
Oct 31 07:27:51.883699 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:51.883666    2577 generic.go:356] "Generic (PLEG): container finished" podID="60bef5910d750df47fec5174fe909f02" containerID="f7e2cee5c0878f4e2d83dec395209141fd051820c005561f91b8a9e8dcf0f019" exitCode=0
Oct 31 07:27:51.883836 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:51.883794    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerDied","Data":"f7e2cee5c0878f4e2d83dec395209141fd051820c005561f91b8a9e8dcf0f019"}
Oct 31 07:27:51.886510 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:51.886485    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pkj5d" event={"ID":"132e4deb-32cc-4b7b-a179-44a9700f1bac","Type":"ContainerStarted","Data":"0d18aeabcd621beea6eb4bf82676d7aaec1eff7b3a0a424d197c41a464069672"}
Oct 31 07:27:51.889097 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:51.889072    2577 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="26a555ea3cfb53e094af4d2c710104a1cac581a108671596c4f49ff4affb1c68" exitCode=0
Oct 31 07:27:51.889176 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:51.889112    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cc6gh" event={"ID":"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb","Type":"ContainerDied","Data":"26a555ea3cfb53e094af4d2c710104a1cac581a108671596c4f49ff4affb1c68"}
Oct 31 07:27:51.918165 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:51.918097    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" podStartSLOduration=6.9180779260000005 podStartE2EDuration="6.918077926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:27:51.917335094 +0000 UTC m=+6.715019228" watchObservedRunningTime="2026-10-31 07:27:51.918077926 +0000 UTC m=+6.715762045"
Oct 31 07:27:52.038381 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:52.038319    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pkj5d" podStartSLOduration=2.601839993 podStartE2EDuration="7.038298162s" totalImagesPullingTime="4.436458169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:27:51.995936199 +0000 UTC m=+6.793620308" watchObservedRunningTime="2026-10-31 07:27:52.038298162 +0000 UTC m=+6.835982276"
Oct 31 07:27:52.738074 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:52.737985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:52.738221 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:52.738098    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:27:52.738430 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:52.738415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:52.738524 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:52.738483    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:27:52.895807 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:52.895768    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wbg8z" event={"ID":"f206a32f-819d-4e61-9040-790e8c9183ec","Type":"ContainerStarted","Data":"4d871b92fc5c82a5d7c1e8cf35e19bbba1138b37cb4754511e9c8d34853cd0d0"}
Oct 31 07:27:52.905016 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:52.904978    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerStarted","Data":"5afc719f006c15a4a9bdfc29b47b7d25b81e1cb14bc0e6113120b60fa0d4b8cd"}
Oct 31 07:27:52.977490 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:52.977425    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wbg8z" podStartSLOduration=2.597558328 podStartE2EDuration="7.977408581s" totalImagesPullingTime="5.379850253s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:27:52.938504726 +0000 UTC m=+7.736188835" watchObservedRunningTime="2026-10-31 07:27:52.977408581 +0000 UTC m=+7.775092705"
Oct 31 07:27:52.977681 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:52.977649    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" podStartSLOduration=7.9776406170000005 podStartE2EDuration="7.977640617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:27:52.975916893 +0000 UTC m=+7.773601006" watchObservedRunningTime="2026-10-31 07:27:52.977640617 +0000 UTC m=+7.775324726"
Oct 31 07:27:53.738497 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:53.738465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:53.738672 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:53.738614    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:27:53.913257 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:53.913222    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl" event={"ID":"db612775-57b2-405c-a403-8c78f173d018","Type":"ContainerStarted","Data":"c3a81971be7e41bf79f71174fadbe91fff51eebbecb6f05eef68e96c651be4c5"}
Oct 31 07:27:54.389594 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:54.389550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:54.389773 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:54.389612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:54.389826 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.389802    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:54.389886 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.389872    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret podName:f0f4a456-26c2-4a45-bd30-b6795d5c03cc nodeName:}" failed. No retries permitted until 2026-10-31 07:28:02.38985279 +0000 UTC m=+17.187536895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret") pod "global-pull-secret-syncer-82457" (UID: "f0f4a456-26c2-4a45-bd30-b6795d5c03cc") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:27:54.390354 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.390330    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:54.390456 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.390394    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs podName:cee91069-0b51-442c-a3c3-71e71302b0e5 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:02.390377373 +0000 UTC m=+17.188061472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs") pod "network-metrics-daemon-s2r9l" (UID: "cee91069-0b51-442c-a3c3-71e71302b0e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:27:54.490946 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:54.490842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:54.491096 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.491014    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:27:54.491096 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.491037    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:27:54.491096 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.491051    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4944 for pod openshift-network-diagnostics/network-check-target-zb9td: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:54.491259 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.491108    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944 podName:8856b22a-b791-4ca1-8083-650b391429af nodeName:}" failed. No retries permitted until 2026-10-31 07:28:02.491091496 +0000 UTC m=+17.288775600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4944" (UniqueName: "kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944") pod "network-check-target-zb9td" (UID: "8856b22a-b791-4ca1-8083-650b391429af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:27:54.689597 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:54.688855    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:54.689597 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:54.689452    2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:27:54.739366 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:54.738698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:54.739366 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.738822    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:27:54.739366 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:54.739203    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:54.739366 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:54.739316    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:27:54.867853 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:54.867836    2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 07:27:54.919694 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:54.919593    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl" event={"ID":"db612775-57b2-405c-a403-8c78f173d018","Type":"ContainerStarted","Data":"c2d95e8cc4202a6c21b5ee585a352268997e7ee84ce4a606fb0f668434024413"}
Oct 31 07:27:55.690042 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:55.688870    2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:27:54.867850858Z","UUID":"b16d2e6e-a657-46d1-b7de-240455de0c77","Handler":null,"Name":"","Endpoint":""}
Oct 31 07:27:55.691396 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:55.691274    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 07:27:55.691396 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:55.691331    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 07:27:55.739585 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:55.739554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:55.739915 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:55.739889    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:27:55.925006 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:55.924965    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v59hm" event={"ID":"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7","Type":"ContainerStarted","Data":"cd4e660ac560b829d9920de0ad49ca5335de02723cdeb179572f893699393d40"}
Oct 31 07:27:55.927954 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:55.927921    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5jj86" event={"ID":"1e3a7ead-4527-476b-9b02-c50e68e80166","Type":"ContainerStarted","Data":"b480aa047be27f5c1c2069ab05defe441919e97d6d81b0367e84690fb4058b60"}
Oct 31 07:27:55.964491 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:55.964439    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-v59hm" podStartSLOduration=2.643989652 podStartE2EDuration="10.964422377s" totalImagesPullingTime="8.320432725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:27:55.963611071 +0000 UTC m=+10.761295180" watchObservedRunningTime="2026-10-31 07:27:55.964422377 +0000 UTC m=+10.762106487"
Oct 31 07:27:56.738729 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:56.738533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:56.738923 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:56.738533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:56.738923 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:56.738838    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:27:56.739089 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:56.738932    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:27:56.932823 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:56.932785    2577 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="441ab54ecf9418722396958c40f2d03efebede35ffcede7a2e6a5728f9a102f2" exitCode=0
Oct 31 07:27:56.933262 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:56.933074    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cc6gh" event={"ID":"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb","Type":"ContainerDied","Data":"441ab54ecf9418722396958c40f2d03efebede35ffcede7a2e6a5728f9a102f2"}
Oct 31 07:27:56.935685 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:56.935554    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-96gd4" event={"ID":"2af799e3-eb79-41f6-be34-411ea2f10326","Type":"ContainerStarted","Data":"f22d675ac174b7a2ae483efced4d54c7d5e7c0b6d1f9297e37524f5119a86d61"}
Oct 31 07:27:56.938259 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:56.938237    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl" event={"ID":"db612775-57b2-405c-a403-8c78f173d018","Type":"ContainerStarted","Data":"ef9b6ced3c8691ee25a03c7bdf14a38edbeb45347c3697b3ff188b1892f8fad0"}
Oct 31 07:27:57.001864 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:57.001767    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5jj86" podStartSLOduration=3.862047677 podStartE2EDuration="12.001753425s" totalImagesPullingTime="8.139705748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:27:56.004621292 +0000 UTC m=+10.802305399" watchObservedRunningTime="2026-10-31 07:27:57.001753425 +0000 UTC m=+11.799437532"
Oct 31 07:27:57.028182 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:57.028125    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl" podStartSLOduration=3.326325833 podStartE2EDuration="12.028106457s" totalImagesPullingTime="8.701780624s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:27:57.027268764 +0000 UTC m=+11.824952871" watchObservedRunningTime="2026-10-31 07:27:57.028106457 +0000 UTC m=+11.825790567"
Oct 31 07:27:57.074043 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:57.073985    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-96gd4" podStartSLOduration=3.692613337 podStartE2EDuration="12.073968415s" totalImagesPullingTime="8.381355078s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:27:57.064325389 +0000 UTC m=+11.862009498" watchObservedRunningTime="2026-10-31 07:27:57.073968415 +0000 UTC m=+11.871652522"
Oct 31 07:27:57.738551 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:57.738478    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:57.738686 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:57.738624    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:27:57.941886 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:57.941838    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-86nzk" event={"ID":"836cf52b-b0e8-4e98-a812-21d0cebfaf63","Type":"ContainerStarted","Data":"e48d7ea3d6ab5152c05557020f7d211f40bacd277808c8e6e0055b63f97dc654"}
Oct 31 07:27:57.944630 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:57.944598    2577 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="e6d10b6029d7e61aca133c7bf586726749005f14c5c0f73499dbf9c65c23b844" exitCode=0
Oct 31 07:27:57.944763 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:57.944680    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cc6gh" event={"ID":"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb","Type":"ContainerDied","Data":"e6d10b6029d7e61aca133c7bf586726749005f14c5c0f73499dbf9c65c23b844"}
Oct 31 07:27:57.986021 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:57.985958    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-86nzk" podStartSLOduration=2.742861425 podStartE2EDuration="12.985940059s" totalImagesPullingTime="10.243078634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:27:57.985606207 +0000 UTC m=+12.783290314" watchObservedRunningTime="2026-10-31 07:27:57.985940059 +0000 UTC m=+12.783624168"
Oct 31 07:27:58.738847 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:58.738809    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:27:58.739023 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:58.738942    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:27:58.739023 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:58.738999    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:27:58.739141 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:58.739115    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:27:59.738048 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:59.738006    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:27:59.738705 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:27:59.738168    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:27:59.952000 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:59.951962    2577 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="d4b0181eafb80a7a545229514c8226f04d227637373b6fa92d84567b01eed01e" exitCode=0
Oct 31 07:27:59.952153 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:27:59.952024    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cc6gh" event={"ID":"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb","Type":"ContainerDied","Data":"d4b0181eafb80a7a545229514c8226f04d227637373b6fa92d84567b01eed01e"}
Oct 31 07:28:00.738809 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:00.738573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:00.739341 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:00.738909    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:00.739341 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:00.738663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:00.739341 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:00.739003    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:01.737923 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:01.737893    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:01.738069 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:01.738006    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:02.458111 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.458076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:02.458453 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.458121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:02.458453 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.458213    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:02.458453 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.458271    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret podName:f0f4a456-26c2-4a45-bd30-b6795d5c03cc nodeName:}" failed. No retries permitted until 2026-10-31 07:28:18.458250493 +0000 UTC m=+33.255934578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret") pod "global-pull-secret-syncer-82457" (UID: "f0f4a456-26c2-4a45-bd30-b6795d5c03cc") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:02.458453 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.458213    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:02.458453 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.458332    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs podName:cee91069-0b51-442c-a3c3-71e71302b0e5 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:18.458322754 +0000 UTC m=+33.256006853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs") pod "network-metrics-daemon-s2r9l" (UID: "cee91069-0b51-442c-a3c3-71e71302b0e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:02.558970 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.558937    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:02.559129 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.559108    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:02.559195 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.559136    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:02.559195 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.559150    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4944 for pod openshift-network-diagnostics/network-check-target-zb9td: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:02.559380 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.559202    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944 podName:8856b22a-b791-4ca1-8083-650b391429af nodeName:}" failed. No retries permitted until 2026-10-31 07:28:18.559187427 +0000 UTC m=+33.356871537 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4944" (UniqueName: "kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944") pod "network-check-target-zb9td" (UID: "8856b22a-b791-4ca1-8083-650b391429af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:02.738625 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.738446    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:02.738773 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.738475    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:02.738773 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.738737    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:02.738899 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:02.738779    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:02.960762 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.960672    2577 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="4265ed8ba65349630249001f218f5905b09a361ebebaba3c4f149bd117576afc" exitCode=0
Oct 31 07:28:02.960900 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.960765    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cc6gh" event={"ID":"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb","Type":"ContainerDied","Data":"4265ed8ba65349630249001f218f5905b09a361ebebaba3c4f149bd117576afc"}
Oct 31 07:28:02.963573 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.963552    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:28:02.963928 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.963903    2577 generic.go:356] "Generic (PLEG): container finished" podID="4fc187c9-2163-4de8-afbb-77f65de777c8" containerID="7adce90669d2aad149ed39c047a8cc97f157b1cdd1dc4e112adb771e5e397eec" exitCode=1
Oct 31 07:28:02.964008 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.963944    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerStarted","Data":"e10129258f0078054bfd88faed8cc6f215ae76ca9340a5f924b8db403f024e18"}
Oct 31 07:28:02.964069 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.964020    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerStarted","Data":"6971cd30e42a2fbf91e11e832086775177bdd6e978642c6c79563423738f2db8"}
Oct 31 07:28:02.964108 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.964070    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerStarted","Data":"6ea98570cf491b8aead3488bf437a66bc62c899c2fab355b90cecf65bbe8bd49"}
Oct 31 07:28:02.964140 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.964117    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerStarted","Data":"4bf4dc5b2697035e77313b8cc7aa3dcda96a887f25ef7586733b27357f052826"}
Oct 31 07:28:02.964177 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.964164    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerDied","Data":"7adce90669d2aad149ed39c047a8cc97f157b1cdd1dc4e112adb771e5e397eec"}
Oct 31 07:28:02.964227 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:02.964214    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerStarted","Data":"b0c16ecbd6b3a1b41538c81dada0e9ad67e318f2556477d7f644c6752c2e3ef8"}
Oct 31 07:28:03.738057 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:03.738021    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:03.738547 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:03.738143    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:03.968893 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:03.968861    2577 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="2c4b8aea403e45be5e7fe64e0e799285abb0e7b5fbd7aa939a177bddd02566b1" exitCode=0
Oct 31 07:28:03.969033 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:03.968894    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cc6gh" event={"ID":"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb","Type":"ContainerDied","Data":"2c4b8aea403e45be5e7fe64e0e799285abb0e7b5fbd7aa939a177bddd02566b1"}
Oct 31 07:28:04.738872 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:04.738841    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:04.739315 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:04.738943    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:04.739315 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:04.738993    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:04.739315 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:04.739088    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:04.793295 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:04.793256    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v59hm_ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7/dns-node-resolver/0.log"
Oct 31 07:28:04.974689 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:04.974607    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cc6gh" event={"ID":"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb","Type":"ContainerStarted","Data":"5ae52c5d0ded4767e761481b09f42988403a31f3b7afe24bf5ea39503be3b630"}
Oct 31 07:28:04.977474 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:04.977454    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:28:04.977759 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:04.977740    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerStarted","Data":"0edd2116c7587ce85943f5765cd740d1fb5cf68f059e1236b426001326a1b2b4"}
Oct 31 07:28:05.047752 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:05.047704    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cc6gh" podStartSLOduration=9.67956108 podStartE2EDuration="20.047690411s" totalImagesPullingTime="10.368129331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:05.047517843 +0000 UTC m=+19.845201951" watchObservedRunningTime="2026-10-31 07:28:05.047690411 +0000 UTC m=+19.845374519"
Oct 31 07:28:05.739237 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:05.739202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:05.739741 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:05.739343    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:05.745332 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:05.745310    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wbg8z_f206a32f-819d-4e61-9040-790e8c9183ec/node-ca/0.log"
Oct 31 07:28:06.738717 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:06.738685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:06.738717 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:06.738717    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:06.738887 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:06.738783    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:06.738887 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:06.738859    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:06.985628 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:06.985599    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:28:06.986315 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:06.985959    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerStarted","Data":"0a296afcf272ba3ea3181342a5f7cdf84299a63d8121d852e87b87621a351d1f"}
Oct 31 07:28:06.986376 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:06.986314    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:06.986462 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:06.986444    2577 scope.go:122] "RemoveContainer" containerID="7adce90669d2aad149ed39c047a8cc97f157b1cdd1dc4e112adb771e5e397eec"
Oct 31 07:28:07.002943 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:07.002920    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:07.738747 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:07.738712    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:07.738982 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:07.738820    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:07.994429 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:07.994357    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:28:07.994849 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:07.994688    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" event={"ID":"4fc187c9-2163-4de8-afbb-77f65de777c8","Type":"ContainerStarted","Data":"6cdd57e5d575de22c5475d6ef1d65ad95652caaffd4243076011a65912693d22"}
Oct 31 07:28:07.994849 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:07.994785    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:28:07.995040 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:07.995027    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:08.010201 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:08.010177    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:08.056334 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:08.056271    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" podStartSLOduration=7.970601458 podStartE2EDuration="23.056258046s" totalImagesPullingTime="15.085656588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:08.056046813 +0000 UTC m=+22.853730921" watchObservedRunningTime="2026-10-31 07:28:08.056258046 +0000 UTC m=+22.853942148"
Oct 31 07:28:08.738748 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:08.738712    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:08.738926 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:08.738821    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:08.738926 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:08.738875    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:08.738991 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:08.738960    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:08.996715 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:08.996643    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:28:09.738343 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:09.738309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:09.738514 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:09.738459    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:09.998830 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:09.998759    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:28:10.738201 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:10.738164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:10.738378 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:10.738164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:10.738378 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:10.738276    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:10.738378 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:10.738369    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:11.738577 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:11.738544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:11.738993 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:11.738654    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:12.738174 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:12.738140    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:12.738174 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:12.738174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:12.738489 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:12.738241    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:12.738489 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:12.738310    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:13.737873 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:13.737843    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:13.738378 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:13.737968    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:14.737936 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:14.737901    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:14.738380 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:14.737911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:14.738380 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:14.738020    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:14.738380 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:14.738073    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:15.738568 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:15.738537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:15.738954 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:15.738642    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:15.840241 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:15.840205    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:15.840490 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:15.840473    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:28:15.854626 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:15.854582    2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" podUID="4fc187c9-2163-4de8-afbb-77f65de777c8" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 31 07:28:15.863126 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:15.863091    2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz" podUID="4fc187c9-2163-4de8-afbb-77f65de777c8" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 31 07:28:16.015008 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:16.014928    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:28:16.015156 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:16.015061    2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:28:16.015512 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:16.015497    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:28:16.738207 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:16.738166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:16.738207 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:16.738196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:16.738497 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:16.738300    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:16.738497 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:16.738413    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:17.738504 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:17.738467    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:17.738873 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:17.738576    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:18.471770 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:18.471735    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:18.471770 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:18.471771    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:18.472017 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.471884    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:18.472017 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.471893    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:18.472017 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.471937    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret podName:f0f4a456-26c2-4a45-bd30-b6795d5c03cc nodeName:}" failed. No retries permitted until 2026-10-31 07:28:50.471924732 +0000 UTC m=+65.269608831 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret") pod "global-pull-secret-syncer-82457" (UID: "f0f4a456-26c2-4a45-bd30-b6795d5c03cc") : object "kube-system"/"original-pull-secret" not registered
Oct 31 07:28:18.472017 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.471950    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs podName:cee91069-0b51-442c-a3c3-71e71302b0e5 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:50.471944434 +0000 UTC m=+65.269628521 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs") pod "network-metrics-daemon-s2r9l" (UID: "cee91069-0b51-442c-a3c3-71e71302b0e5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 07:28:18.572538 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:18.572500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:18.572702 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.572654    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 07:28:18.572702 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.572674    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 07:28:18.572702 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.572688    2577 projected.go:196] Error preparing data for projected volume kube-api-access-h4944 for pod openshift-network-diagnostics/network-check-target-zb9td: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:18.572802 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.572735    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944 podName:8856b22a-b791-4ca1-8083-650b391429af nodeName:}" failed. No retries permitted until 2026-10-31 07:28:50.57272349 +0000 UTC m=+65.370407579 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h4944" (UniqueName: "kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944") pod "network-check-target-zb9td" (UID: "8856b22a-b791-4ca1-8083-650b391429af") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 07:28:18.738748 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:18.738669    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:18.739118 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:18.738675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:18.739118 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.738772    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:18.739118 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:18.738848    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:19.738390 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:19.738362    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:19.738547 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:19.738475    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:20.737988 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:20.737950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:20.737988 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:20.737974    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:20.738467 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:20.738054    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:20.738467 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:20.738169    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:21.738065 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:21.738032    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:21.738488 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:21.738144    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:22.738588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:22.738549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:22.738588 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:22.738579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:22.739040 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:22.738657    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:22.739040 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:22.738775    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:23.738407 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:23.738371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:23.738582 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:23.738523    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:24.738178 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:24.738139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:24.738607 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:24.738250    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:24.738607 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:24.738297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:24.738607 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:24.738384    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:25.739435 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:25.739246    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:25.739786 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:25.739509    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:25.908557 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:25.908512    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82457"]
Oct 31 07:28:25.908887 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:25.908866    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:25.909010 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:25.908987    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:25.911264 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:25.911238    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zb9td"]
Oct 31 07:28:25.911393 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:25.911359    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:25.911470 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:25.911447    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:25.912075 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:25.912055    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s2r9l"]
Oct 31 07:28:26.033438 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:26.033410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:26.033597 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:26.033516    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:27.738656 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:27.738624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:27.738656 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:27.738645    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:27.739272 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:27.738627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:27.739272 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:27.738737    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:27.739272 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:27.738832    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:27.739435 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:27.739306    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:29.738724 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:29.738685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:29.739370 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:29.738744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:29.739370 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:29.738752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:29.739370 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:29.738840    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zb9td" podUID="8856b22a-b791-4ca1-8083-650b391429af"
Oct 31 07:28:29.739370 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:29.738895    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-82457" podUID="f0f4a456-26c2-4a45-bd30-b6795d5c03cc"
Oct 31 07:28:29.739370 ip-10-0-140-236 kubenswrapper[2577]: E1031 07:28:29.738962    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s2r9l" podUID="cee91069-0b51-442c-a3c3-71e71302b0e5"
Oct 31 07:28:29.984913 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:29.984882    2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeReady"
Oct 31 07:28:29.985074 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:29.985012    2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 07:28:30.037900 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.037862    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hzqlg"]
Oct 31 07:28:30.039734 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.039715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.040687 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.040665    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9lgfj"]
Oct 31 07:28:30.044921 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.044893    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:30.057403 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.057380    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-525tp"]
Oct 31 07:28:30.059528 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.059414    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 07:28:30.060388 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.060370    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 07:28:30.060486 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.060440    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vhq6\""
Oct 31 07:28:30.067175 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.067154    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzqlg"]
Oct 31 07:28:30.067306 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.067198    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9lgfj"]
Oct 31 07:28:30.067353 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.067302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.079719 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.079690    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-525tp"]
Oct 31 07:28:30.143959 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.143915    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-cert\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:30.143959 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.143960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457dde04-3e55-4c30-a1c5-d13672f48a4a-tmp-dir\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.144174 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.144034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcflp\" (UniqueName: \"kubernetes.io/projected/457dde04-3e55-4c30-a1c5-d13672f48a4a-kube-api-access-bcflp\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.144174 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.144060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tr9c\" (UniqueName: \"kubernetes.io/projected/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-kube-api-access-6tr9c\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:30.144174 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.144100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/457dde04-3e55-4c30-a1c5-d13672f48a4a-metrics-tls\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.144174 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.144161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/457dde04-3e55-4c30-a1c5-d13672f48a4a-config-volume\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.159846 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.159800    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 07:28:30.160005 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.159888    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dql9j\""
Oct 31 07:28:30.160005 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.159905    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 07:28:30.160005 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.159980    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wkhqf\""
Oct 31 07:28:30.160005 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.159990    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 07:28:30.160144 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.160004    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 07:28:30.160144 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.160031    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 07:28:30.160144 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.160057    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 07:28:30.160144 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.160058    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 07:28:30.245518 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-cert\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:30.245518 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b62296f1-d84a-4dd1-8eb0-9f64567af021-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.245518 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457dde04-3e55-4c30-a1c5-d13672f48a4a-tmp-dir\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.245768 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b62296f1-d84a-4dd1-8eb0-9f64567af021-data-volume\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.245768 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.245768 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcflp\" (UniqueName: \"kubernetes.io/projected/457dde04-3e55-4c30-a1c5-d13672f48a4a-kube-api-access-bcflp\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.245768 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tr9c\" (UniqueName: \"kubernetes.io/projected/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-kube-api-access-6tr9c\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:30.245768 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52lps\" (UniqueName: \"kubernetes.io/projected/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-api-access-52lps\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.245988 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/457dde04-3e55-4c30-a1c5-d13672f48a4a-metrics-tls\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.245988 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b62296f1-d84a-4dd1-8eb0-9f64567af021-crio-socket\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.245988 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/457dde04-3e55-4c30-a1c5-d13672f48a4a-config-volume\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.245988 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.245871    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457dde04-3e55-4c30-a1c5-d13672f48a4a-tmp-dir\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.246427 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.246408    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/457dde04-3e55-4c30-a1c5-d13672f48a4a-config-volume\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.249553 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.249518    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/457dde04-3e55-4c30-a1c5-d13672f48a4a-metrics-tls\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.249683 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.249612    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-cert\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:30.265443 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.265416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcflp\" (UniqueName: \"kubernetes.io/projected/457dde04-3e55-4c30-a1c5-d13672f48a4a-kube-api-access-bcflp\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.265607 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.265587    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tr9c\" (UniqueName: \"kubernetes.io/projected/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-kube-api-access-6tr9c\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:30.347009 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.346970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b62296f1-d84a-4dd1-8eb0-9f64567af021-data-volume\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.347009 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.347012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.347255 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.347036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52lps\" (UniqueName: \"kubernetes.io/projected/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-api-access-52lps\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.347255 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.347086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b62296f1-d84a-4dd1-8eb0-9f64567af021-crio-socket\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.347255 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.347125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b62296f1-d84a-4dd1-8eb0-9f64567af021-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.347437 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.347317    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b62296f1-d84a-4dd1-8eb0-9f64567af021-crio-socket\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.347514 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.347494    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b62296f1-d84a-4dd1-8eb0-9f64567af021-data-volume\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.347632 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.347610    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.349318 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.349302    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b62296f1-d84a-4dd1-8eb0-9f64567af021-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.353327 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.353311    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:30.366720 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.366696    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52lps\" (UniqueName: \"kubernetes.io/projected/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-api-access-52lps\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.373601 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.373582    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:30.379371 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.379256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:30.662552 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.662522    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hzqlg"]
Oct 31 07:28:30.665924 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:30.665897    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457dde04_3e55_4c30_a1c5_d13672f48a4a.slice/crio-943a503a9bc0ee6a374d29505f320454a217981af0d72d26163429f181391af3 WatchSource:0}: Error finding container 943a503a9bc0ee6a374d29505f320454a217981af0d72d26163429f181391af3: Status 404 returned error can't find the container with id 943a503a9bc0ee6a374d29505f320454a217981af0d72d26163429f181391af3
Oct 31 07:28:30.699366 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.699338    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9lgfj"]
Oct 31 07:28:30.702040 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:30.702017    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-525tp"]
Oct 31 07:28:30.702297 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:30.702260    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf1ad8c_64aa_4545_87f3_c4f3f9a916c5.slice/crio-d3667fbda52b872511daaaf50ba7c878c6daadd2b38b2969636c77e8673b2e02 WatchSource:0}: Error finding container d3667fbda52b872511daaaf50ba7c878c6daadd2b38b2969636c77e8673b2e02: Status 404 returned error can't find the container with id d3667fbda52b872511daaaf50ba7c878c6daadd2b38b2969636c77e8673b2e02
Oct 31 07:28:30.705024 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:30.704999    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62296f1_d84a_4dd1_8eb0_9f64567af021.slice/crio-ba447568733a33979f88bcdce24ef1ace1d241fe20e8b32a37135e2d1ba9a0c9 WatchSource:0}: Error finding container ba447568733a33979f88bcdce24ef1ace1d241fe20e8b32a37135e2d1ba9a0c9: Status 404 returned error can't find the container with id ba447568733a33979f88bcdce24ef1ace1d241fe20e8b32a37135e2d1ba9a0c9
Oct 31 07:28:31.046959 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.046860    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-525tp" event={"ID":"b62296f1-d84a-4dd1-8eb0-9f64567af021","Type":"ContainerStarted","Data":"0d052bc3118137a02632bfa3d0c6b475ad0ccc1fe2a846013c70bcd75cb0863c"}
Oct 31 07:28:31.046959 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.046941    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-525tp" event={"ID":"b62296f1-d84a-4dd1-8eb0-9f64567af021","Type":"ContainerStarted","Data":"ba447568733a33979f88bcdce24ef1ace1d241fe20e8b32a37135e2d1ba9a0c9"}
Oct 31 07:28:31.047966 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.047941    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzqlg" event={"ID":"457dde04-3e55-4c30-a1c5-d13672f48a4a","Type":"ContainerStarted","Data":"943a503a9bc0ee6a374d29505f320454a217981af0d72d26163429f181391af3"}
Oct 31 07:28:31.049220 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.049190    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9lgfj" event={"ID":"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5","Type":"ContainerStarted","Data":"d3667fbda52b872511daaaf50ba7c878c6daadd2b38b2969636c77e8673b2e02"}
Oct 31 07:28:31.738431 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.738395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:31.738600 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.738460    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:31.738801 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.738779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:31.754781 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.754755    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 07:28:31.755037 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.754940    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 07:28:31.768190 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.768166    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-548vl\""
Oct 31 07:28:31.768337 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.768232    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nzlr\""
Oct 31 07:28:31.768337 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.768310    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 07:28:31.768436 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.768377    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 07:28:31.781296 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:31.781241    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qk9pc\""
Oct 31 07:28:32.054612 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:32.054513    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-525tp" event={"ID":"b62296f1-d84a-4dd1-8eb0-9f64567af021","Type":"ContainerStarted","Data":"7b120d360d257089d2bcbe721f0daa6acad8d830d6099d31e71471521a76da5f"}
Oct 31 07:28:33.060937 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.060902    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzqlg" event={"ID":"457dde04-3e55-4c30-a1c5-d13672f48a4a","Type":"ContainerStarted","Data":"f31c1cad87ebca14c350ff9c8b5bc54981a39b45e5ca1dc069ee97754b1791a9"}
Oct 31 07:28:33.061410 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.060995    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hzqlg" event={"ID":"457dde04-3e55-4c30-a1c5-d13672f48a4a","Type":"ContainerStarted","Data":"3502d0a3cd28d017117f5e00c5e45e7c9a3b5772cbb0062b006f397946b49dac"}
Oct 31 07:28:33.061410 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.061030    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:33.101998 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.101941    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-hzqlg" podStartSLOduration=1.269562914 podStartE2EDuration="3.101922706s" totalImagesPullingTime="1.832359792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:33.100832058 +0000 UTC m=+47.898516166" watchObservedRunningTime="2026-10-31 07:28:33.101922706 +0000 UTC m=+47.899606816"
Oct 31 07:28:33.599214 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.599130    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"]
Oct 31 07:28:33.602316 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.602300    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.620141 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.620102    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 07:28:33.620337 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.620206    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cg7nk\""
Oct 31 07:28:33.620337 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.620225    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 07:28:33.620337 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.620207    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 07:28:33.620337 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.620256    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 07:28:33.620337 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.620319    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 07:28:33.621081 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.621061    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"]
Oct 31 07:28:33.649270 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.649241    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"]
Oct 31 07:28:33.653267 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.653243    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.669430 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.669397    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hbnc6"]
Oct 31 07:28:33.672442 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.672419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.675242 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.675217    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"]
Oct 31 07:28:33.687076 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.687056    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 07:28:33.687304 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.687269    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 07:28:33.687391 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.687363    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m2lq9\""
Oct 31 07:28:33.725521 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.725491    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 07:28:33.725801 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.725786    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 07:28:33.725889 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.725875    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ppvgk\""
Oct 31 07:28:33.725940 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.725907    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 07:28:33.725976 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.725960    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 07:28:33.775275 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775239    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpnv5\" (UniqueName: \"kubernetes.io/projected/658d33ef-d90b-448b-b4d0-e175b77a187f-kube-api-access-mpnv5\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.775275 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.775474 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775312    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-tls\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.775474 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-root\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.775474 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/658d33ef-d90b-448b-b4d0-e175b77a187f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.775561 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.775561 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-metrics-client-ca\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.775621 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95b1f652-e473-4b0e-a9da-ac32940f95a8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.775621 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.775672 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.775703 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775672    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.775703 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxz6l\" (UniqueName: \"kubernetes.io/projected/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-api-access-kxz6l\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.775758 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-sys\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.775758 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-kube-api-access-cffpp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.775815 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.775848 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-textfile\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.775848 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-wtmp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.775902 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.775902 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.775871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.877070 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.876975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/658d33ef-d90b-448b-b4d0-e175b77a187f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.877070 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.877070 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-metrics-client-ca\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.877377 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95b1f652-e473-4b0e-a9da-ac32940f95a8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.877377 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.877377 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.877377 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.877377 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxz6l\" (UniqueName: \"kubernetes.io/projected/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-api-access-kxz6l\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.877377 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-sys\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.877377 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-kube-api-access-cffpp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.877720 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877631    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95b1f652-e473-4b0e-a9da-ac32940f95a8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.877845 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877818    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-metrics-client-ca\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.877961 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877906    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-sys\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.877961 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877925    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.877961 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877949    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/658d33ef-d90b-448b-b4d0-e175b77a187f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.878079 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.877942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.878109 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878043    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-textfile\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.878142 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878112    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-wtmp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.878142 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878137    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.878254 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.878254 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpnv5\" (UniqueName: \"kubernetes.io/projected/658d33ef-d90b-448b-b4d0-e175b77a187f-kube-api-access-mpnv5\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.878380 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.878380 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-tls\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.878380 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-root\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.878517 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878375    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-textfile\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.878570 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.878553    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-wtmp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.879853 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.879824    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-root\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.880719 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.880696    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.880944 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.880880    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.881054 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.880989    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.884164 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.881191    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.884164 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.881262    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.884164 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.881323    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-tls\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.884164 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.881623    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.886508 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.886487    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.898021 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.897986    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-kube-api-access-cffpp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:33.898143 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.898124    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxz6l\" (UniqueName: \"kubernetes.io/projected/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-api-access-kxz6l\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.898241 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.898214    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpnv5\" (UniqueName: \"kubernetes.io/projected/658d33ef-d90b-448b-b4d0-e175b77a187f-kube-api-access-mpnv5\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.917075 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.917044    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:33.970639 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.970607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:33.992498 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:33.990660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:34.008756 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:34.008726    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3618d95f_af7c_4e0c_8ddd_3a5bbd18e2ee.slice/crio-70f074b267a802533fc01318aebb2b926b621bfa14a701947086928a0c04e779 WatchSource:0}: Error finding container 70f074b267a802533fc01318aebb2b926b621bfa14a701947086928a0c04e779: Status 404 returned error can't find the container with id 70f074b267a802533fc01318aebb2b926b621bfa14a701947086928a0c04e779
Oct 31 07:28:34.068437 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:34.068401    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-525tp" event={"ID":"b62296f1-d84a-4dd1-8eb0-9f64567af021","Type":"ContainerStarted","Data":"2bb01087636ffa945978b98170de0f3410a6d86ae6ebca84fe8bc3115d234bf6"}
Oct 31 07:28:34.069599 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:34.069575    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbnc6" event={"ID":"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee","Type":"ContainerStarted","Data":"70f074b267a802533fc01318aebb2b926b621bfa14a701947086928a0c04e779"}
Oct 31 07:28:34.070887 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:34.070864    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9lgfj" event={"ID":"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5","Type":"ContainerStarted","Data":"ddb07a6d4917d58638951a9a5da6681bf7cf475edca52f221e575bf9448944b8"}
Oct 31 07:28:34.109274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:34.109220    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-525tp" podStartSLOduration=1.890555416 podStartE2EDuration="4.109203752s" totalImagesPullingTime="2.218648336s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:34.108259332 +0000 UTC m=+48.905943441" watchObservedRunningTime="2026-10-31 07:28:34.109203752 +0000 UTC m=+48.906887860"
Oct 31 07:28:34.146833 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:34.146781    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9lgfj" podStartSLOduration=1.5484445820000001 podStartE2EDuration="4.146765929s" totalImagesPullingTime="2.598321347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:34.14585567 +0000 UTC m=+48.943539778" watchObservedRunningTime="2026-10-31 07:28:34.146765929 +0000 UTC m=+48.944450037"
Oct 31 07:28:34.225935 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:34.225911    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"]
Oct 31 07:28:34.228121 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:34.228095    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod658d33ef_d90b_448b_b4d0_e175b77a187f.slice/crio-8401c67bdc047a50ee8712abf8fc395becfaa60b763a3985fab0d4d539031cc9 WatchSource:0}: Error finding container 8401c67bdc047a50ee8712abf8fc395becfaa60b763a3985fab0d4d539031cc9: Status 404 returned error can't find the container with id 8401c67bdc047a50ee8712abf8fc395becfaa60b763a3985fab0d4d539031cc9
Oct 31 07:28:34.290855 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:34.290830    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"]
Oct 31 07:28:34.293223 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:34.293173    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b1f652_e473_4b0e_a9da_ac32940f95a8.slice/crio-c24134f727811f6856dab38b787722771adb75015ff707ed8b73e7e7ecfe53f2 WatchSource:0}: Error finding container c24134f727811f6856dab38b787722771adb75015ff707ed8b73e7e7ecfe53f2: Status 404 returned error can't find the container with id c24134f727811f6856dab38b787722771adb75015ff707ed8b73e7e7ecfe53f2
Oct 31 07:28:35.071156 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.071126    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 07:28:35.076170 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.076146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.076298 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.076224    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf" event={"ID":"658d33ef-d90b-448b-b4d0-e175b77a187f","Type":"ContainerStarted","Data":"2ce2dcbdb92ba4631597a799e475f8b9310b705a707e577e9730c71d00118597"}
Oct 31 07:28:35.076298 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.076270    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf" event={"ID":"658d33ef-d90b-448b-b4d0-e175b77a187f","Type":"ContainerStarted","Data":"b8984199c8ad67b678088d056756e686e2780448582342d91cedff846e510258"}
Oct 31 07:28:35.076411 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.076319    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf" event={"ID":"658d33ef-d90b-448b-b4d0-e175b77a187f","Type":"ContainerStarted","Data":"8401c67bdc047a50ee8712abf8fc395becfaa60b763a3985fab0d4d539031cc9"}
Oct 31 07:28:35.079117 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.078663    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf" event={"ID":"95b1f652-e473-4b0e-a9da-ac32940f95a8","Type":"ContainerStarted","Data":"c24134f727811f6856dab38b787722771adb75015ff707ed8b73e7e7ecfe53f2"}
Oct 31 07:28:35.092024 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.091990    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 07:28:35.092728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.092392    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 07:28:35.092728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.092487    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 07:28:35.092728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.092500    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 07:28:35.092728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.092501    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 07:28:35.092728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.092560    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 07:28:35.092728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.092582    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 07:28:35.092728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.092600    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 07:28:35.092728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.092604    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6pt6b\""
Oct 31 07:28:35.092728 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.092684    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 07:28:35.102493 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.102461    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 07:28:35.201234 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201469 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-web-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201469 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201469 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201625 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201625 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201625 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-out\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201625 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201803 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2wzb\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-kube-api-access-x2wzb\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201803 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201803 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201803 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.201972 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.201949    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.303356 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.303319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-web-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.303533 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.303371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.303533 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.303413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.303533 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.303432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.303533 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.303451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.303912 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.303876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-out\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.304021 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.303907    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.304021 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.303944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.304021 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.303973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2wzb\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-kube-api-access-x2wzb\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.304021 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.304005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.304259 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.304034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.304259 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.304058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.304259 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.304082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.304259 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.304131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.305781 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.304498    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.305781 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.305311    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.306598 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.306575    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.307118 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.307046    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-web-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.307644 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.307601    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.307721 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.307650    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.307721 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.307709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.308714 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.308697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.309047 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.309032    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.309507 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.309485    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.309727 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.309704    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-out\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.325936 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.325908    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2wzb\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-kube-api-access-x2wzb\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.425437 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.425395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:35.798133 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.798104    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 07:28:35.801271 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:35.801239    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fd0630_ea16_413c_a195_ff4cd58d8d21.slice/crio-e9d25f1c3282d0520135486e3cfbe329989f8d3152940e6988a10910b6e59f4c WatchSource:0}: Error finding container e9d25f1c3282d0520135486e3cfbe329989f8d3152940e6988a10910b6e59f4c: Status 404 returned error can't find the container with id e9d25f1c3282d0520135486e3cfbe329989f8d3152940e6988a10910b6e59f4c
Oct 31 07:28:35.899704 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.899673    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8cd777578-wbgnk"]
Oct 31 07:28:35.903439 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.903418    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.909003 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.908977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-grpc-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.909108 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.909030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-tmp\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.909145 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.909118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.909193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.909146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.909193 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.909166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.909274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.909196    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.909274 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.909235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgglw\" (UniqueName: \"kubernetes.io/projected/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-kube-api-access-hgglw\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.909429 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.909407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-metrics-client-ca\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.909510 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.909488    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:35.922405 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.922376    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 07:28:35.925820 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.925795    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 07:28:35.926729 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.925842    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 07:28:35.926729 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.926070    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 07:28:35.926729 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.926168    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fdnb6\""
Oct 31 07:28:35.926729 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.926246    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 07:28:35.926729 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.926551    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8cd777578-wbgnk"]
Oct 31 07:28:35.927425 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:35.927401    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-78uabpe1lkr2i\""
Oct 31 07:28:36.010109 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.010345 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-grpc-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.010345 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-tmp\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.010345 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010185    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.010345 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.010345 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010222    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.010345 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.010345 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgglw\" (UniqueName: \"kubernetes.io/projected/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-kube-api-access-hgglw\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.010345 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-metrics-client-ca\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.011074 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.010750    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-tmp\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.011835 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.011802    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-metrics-client-ca\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.013064 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.013040    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-grpc-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.013195 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.013171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.013898 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.013255    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.013898 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.013314    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.013898 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.013856    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.014086 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.013897    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.032093 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.032066    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgglw\" (UniqueName: \"kubernetes.io/projected/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-kube-api-access-hgglw\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.084214 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.084167    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf" event={"ID":"658d33ef-d90b-448b-b4d0-e175b77a187f","Type":"ContainerStarted","Data":"2307eab1858e8b4e1861177c4ad319fe15ebd9c9800c41451784c14640534900"}
Oct 31 07:28:36.086596 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.086563    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf" event={"ID":"95b1f652-e473-4b0e-a9da-ac32940f95a8","Type":"ContainerStarted","Data":"5ffde6459d918755d2dd6f1d898c9c870bb799b7690119ff2eb9ce61a9d5a84c"}
Oct 31 07:28:36.086731 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.086623    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf" event={"ID":"95b1f652-e473-4b0e-a9da-ac32940f95a8","Type":"ContainerStarted","Data":"83287250ae5e52bc6b70da938d4f712f7bfa531bccbe380a743990ad444cae82"}
Oct 31 07:28:36.086731 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.086651    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf" event={"ID":"95b1f652-e473-4b0e-a9da-ac32940f95a8","Type":"ContainerStarted","Data":"0fb74f3785e75b8629411a697935a25e0069a3d0519ab1f3321a64c5ae2b8074"}
Oct 31 07:28:36.088197 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.088173    2577 generic.go:356] "Generic (PLEG): container finished" podID="3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee" containerID="19224fd67bb411f5258a5a864f86fc687c3750b9a104f5eb426cb90a76f2c257" exitCode=0
Oct 31 07:28:36.088320 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.088240    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbnc6" event={"ID":"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee","Type":"ContainerDied","Data":"19224fd67bb411f5258a5a864f86fc687c3750b9a104f5eb426cb90a76f2c257"}
Oct 31 07:28:36.089420 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.089399    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fd0630-ea16-413c-a195-ff4cd58d8d21","Type":"ContainerStarted","Data":"e9d25f1c3282d0520135486e3cfbe329989f8d3152940e6988a10910b6e59f4c"}
Oct 31 07:28:36.124712 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.124661    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf" podStartSLOduration=2.134363355 podStartE2EDuration="3.124646871s" totalImagesPullingTime="990.283516ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:36.123720428 +0000 UTC m=+50.921404537" watchObservedRunningTime="2026-10-31 07:28:36.124646871 +0000 UTC m=+50.922330978"
Oct 31 07:28:36.170237 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.170185    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf" podStartSLOduration=1.977550921 podStartE2EDuration="3.170171234s" totalImagesPullingTime="1.192620313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:36.166318353 +0000 UTC m=+50.964002484" watchObservedRunningTime="2026-10-31 07:28:36.170171234 +0000 UTC m=+50.967855339"
Oct 31 07:28:36.226576 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.226547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:36.535772 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:36.535681    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8cd777578-wbgnk"]
Oct 31 07:28:36.538832 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:36.538798    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf252c1a4_f927_4c73_9521_93f5cfbf2dc4.slice/crio-da6208239c5f55b66611f1e47bb739d135e8abef821c0c42f3919c7c468f978e WatchSource:0}: Error finding container da6208239c5f55b66611f1e47bb739d135e8abef821c0c42f3919c7c468f978e: Status 404 returned error can't find the container with id da6208239c5f55b66611f1e47bb739d135e8abef821c0c42f3919c7c468f978e
Oct 31 07:28:37.095712 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:37.095602    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk" event={"ID":"f252c1a4-f927-4c73-9521-93f5cfbf2dc4","Type":"ContainerStarted","Data":"da6208239c5f55b66611f1e47bb739d135e8abef821c0c42f3919c7c468f978e"}
Oct 31 07:28:37.098984 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:37.098743    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbnc6" event={"ID":"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee","Type":"ContainerStarted","Data":"e61cf27e49b9c908ef497ac7f5ec1471af4eb94d336493eaa0d322d2a88ce901"}
Oct 31 07:28:37.098984 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:37.098801    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbnc6" event={"ID":"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee","Type":"ContainerStarted","Data":"405aca7678f8fbb8147a99255dc03851d912a620b3419ab1367a218677dec77f"}
Oct 31 07:28:37.100945 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:37.100836    2577 generic.go:356] "Generic (PLEG): container finished" podID="c0fd0630-ea16-413c-a195-ff4cd58d8d21" containerID="2d31e1e743d4a40168f2f592ffbcf2761d4957f3e2cd7b79fb3172f3d43a26da" exitCode=0
Oct 31 07:28:37.100945 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:37.100940    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fd0630-ea16-413c-a195-ff4cd58d8d21","Type":"ContainerDied","Data":"2d31e1e743d4a40168f2f592ffbcf2761d4957f3e2cd7b79fb3172f3d43a26da"}
Oct 31 07:28:37.142964 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:37.142912    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hbnc6" podStartSLOduration=3.013930154 podStartE2EDuration="4.142898637s" totalImagesPullingTime="1.048343303s" totalInitContainerRuntime="80.62518ms" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:37.142485847 +0000 UTC m=+51.940169955" watchObservedRunningTime="2026-10-31 07:28:37.142898637 +0000 UTC m=+51.940582745"
Oct 31 07:28:38.108055 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.108015    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b67d5b6c9-897nz"]
Oct 31 07:28:38.111291 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.111259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.128664 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.128629    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 07:28:38.128664 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.128638    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 07:28:38.131967 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.131939    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-client-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.132108 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.132086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.132156 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.132139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfg4b\" (UniqueName: \"kubernetes.io/projected/c516d07a-8d53-4cae-9f4e-8b07114916b8-kube-api-access-qfg4b\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.132213 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.132198    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-metrics-server-audit-profiles\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.132416 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.132392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-client-certs\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.132530 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.132454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c516d07a-8d53-4cae-9f4e-8b07114916b8-audit-log\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.132530 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.132507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-tls\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.143667 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.143639    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b67d5b6c9-897nz"]
Oct 31 07:28:38.207447 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.207416    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 07:28:38.207447 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.207440    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1lcltrbisfsua\""
Oct 31 07:28:38.207671 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.207457    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 07:28:38.207671 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.207533    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-58mwv\""
Oct 31 07:28:38.234128 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.234087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c516d07a-8d53-4cae-9f4e-8b07114916b8-audit-log\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.234346 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.234148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-tls\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.234346 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.234181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-client-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.234346 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.234231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.234346 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.234270    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfg4b\" (UniqueName: \"kubernetes.io/projected/c516d07a-8d53-4cae-9f4e-8b07114916b8-kube-api-access-qfg4b\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.234346 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.234322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-metrics-server-audit-profiles\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.234606 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.234373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-client-certs\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.234643 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.234598    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c516d07a-8d53-4cae-9f4e-8b07114916b8-audit-log\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.235252 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.235224    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.235443 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.235358    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-metrics-server-audit-profiles\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.237450 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.237428    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-client-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.237871 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.237834    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-tls\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.241936 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.241913    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-client-certs\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.254250 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.254215    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfg4b\" (UniqueName: \"kubernetes.io/projected/c516d07a-8d53-4cae-9f4e-8b07114916b8-kube-api-access-qfg4b\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.426220 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.426187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:38.476716 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.476688    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"]
Oct 31 07:28:38.481648 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.481626    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:38.497307 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.497115    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"]
Oct 31 07:28:38.537399 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.537371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6l87\" (UniqueName: \"kubernetes.io/projected/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-kube-api-access-w6l87\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:38.537538 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.537423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:38.561791 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.561759    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 07:28:38.561915 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.561831    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8qb8s\""
Oct 31 07:28:38.638847 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.638437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6l87\" (UniqueName: \"kubernetes.io/projected/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-kube-api-access-w6l87\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:38.638847 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.638517    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:38.640774 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.640746    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:38.658841 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.658469    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6l87\" (UniqueName: \"kubernetes.io/projected/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-kube-api-access-w6l87\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:38.779650 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.779621    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b67d5b6c9-897nz"]
Oct 31 07:28:38.782946 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:38.782907    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc516d07a_8d53_4cae_9f4e_8b07114916b8.slice/crio-ed8c10ee633e1ad18b4096e6c91ded20f77169bd3a89deab0b0c90b981179749 WatchSource:0}: Error finding container ed8c10ee633e1ad18b4096e6c91ded20f77169bd3a89deab0b0c90b981179749: Status 404 returned error can't find the container with id ed8c10ee633e1ad18b4096e6c91ded20f77169bd3a89deab0b0c90b981179749
Oct 31 07:28:38.799724 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:38.799617    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:39.100796 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.100620    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"]
Oct 31 07:28:39.104077 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:39.104047    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7a7365_f2ec_4b65_b8ce_4e7ad72aa7ea.slice/crio-db53f4ce4263659743fb99007a507a0a5ee3401bf56259ba3de1b22515d21de0 WatchSource:0}: Error finding container db53f4ce4263659743fb99007a507a0a5ee3401bf56259ba3de1b22515d21de0: Status 404 returned error can't find the container with id db53f4ce4263659743fb99007a507a0a5ee3401bf56259ba3de1b22515d21de0
Oct 31 07:28:39.111590 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.111559    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk" event={"ID":"f252c1a4-f927-4c73-9521-93f5cfbf2dc4","Type":"ContainerStarted","Data":"e9663c0dc813c361f85bd60f16a03045c7e686f2946c2f0e040a8e3e6859ad0d"}
Oct 31 07:28:39.111947 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.111646    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk" event={"ID":"f252c1a4-f927-4c73-9521-93f5cfbf2dc4","Type":"ContainerStarted","Data":"33bbf0a00812a3786edd2c42dd1756cc068ef752df0e3eca90491cc3d5fc31cb"}
Oct 31 07:28:39.111947 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.111682    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk" event={"ID":"f252c1a4-f927-4c73-9521-93f5cfbf2dc4","Type":"ContainerStarted","Data":"b1fc4f5e04aa06737751cff44b513dc51c407e3fe6649e987899569814338361"}
Oct 31 07:28:39.112882 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.112855    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t" event={"ID":"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea","Type":"ContainerStarted","Data":"db53f4ce4263659743fb99007a507a0a5ee3401bf56259ba3de1b22515d21de0"}
Oct 31 07:28:39.114184 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.114158    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz" event={"ID":"c516d07a-8d53-4cae-9f4e-8b07114916b8","Type":"ContainerStarted","Data":"ed8c10ee633e1ad18b4096e6c91ded20f77169bd3a89deab0b0c90b981179749"}
Oct 31 07:28:39.117779 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.117759    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fd0630-ea16-413c-a195-ff4cd58d8d21","Type":"ContainerStarted","Data":"ed8c153f855a0dc498019d05881e21e37af45ab8e19c81bec3e73911b4756a8d"}
Oct 31 07:28:39.117876 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.117806    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fd0630-ea16-413c-a195-ff4cd58d8d21","Type":"ContainerStarted","Data":"0055312547a19932efb7c43f70399b14f8bb7617ab780632890ffbdd66df68e8"}
Oct 31 07:28:39.117876 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.117831    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fd0630-ea16-413c-a195-ff4cd58d8d21","Type":"ContainerStarted","Data":"8f37dacbf438cf1e3db0d71daffad6a580f290511972d0b05e57a196d8ae93cc"}
Oct 31 07:28:39.117876 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.117852    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fd0630-ea16-413c-a195-ff4cd58d8d21","Type":"ContainerStarted","Data":"8aa77838bde75a8c9283b9087cefc1f9340a874983377b18e8934e79949b0351"}
Oct 31 07:28:39.117876 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:39.117873    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fd0630-ea16-413c-a195-ff4cd58d8d21","Type":"ContainerStarted","Data":"f2dd48c07a63758485c0deda4fc68ecd5bde57a39772d26ec863608941011047"}
Oct 31 07:28:40.126715 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.126687    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c0fd0630-ea16-413c-a195-ff4cd58d8d21","Type":"ContainerStarted","Data":"12362413d205c8f0115bc8df8d1238465643dbd9e8fd9005b4ff01a221231b34"}
Oct 31 07:28:40.129973 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.129949    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk" event={"ID":"f252c1a4-f927-4c73-9521-93f5cfbf2dc4","Type":"ContainerStarted","Data":"695a33f94009e384a6d03151366192a61817d49f205b644356104c541d1d2912"}
Oct 31 07:28:40.130067 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.130011    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk" event={"ID":"f252c1a4-f927-4c73-9521-93f5cfbf2dc4","Type":"ContainerStarted","Data":"0ec30fd8656e5ce367b9c448b24d6bf1d4eb5ccbaa1aa73fac33a6b686529ce1"}
Oct 31 07:28:40.130067 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.130039    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk" event={"ID":"f252c1a4-f927-4c73-9521-93f5cfbf2dc4","Type":"ContainerStarted","Data":"a397caae328c81d2436b941731302c030318f918862661216a3c2c118a90efc0"}
Oct 31 07:28:40.130177 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.130095    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:40.214314 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.213756    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.184663534 podStartE2EDuration="5.213735624s" totalImagesPullingTime="3.015925911s" totalInitContainerRuntime="13.146179ms" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:40.199406824 +0000 UTC m=+54.997090932" watchObservedRunningTime="2026-10-31 07:28:40.213735624 +0000 UTC m=+55.011419736"
Oct 31 07:28:40.243673 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.243611    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk" podStartSLOduration=2.40012769 podStartE2EDuration="5.243589526s" totalImagesPullingTime="2.843461836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:40.243142646 +0000 UTC m=+55.040826755" watchObservedRunningTime="2026-10-31 07:28:40.243589526 +0000 UTC m=+55.041273634"
Oct 31 07:28:40.610947 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.610913    2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:28:40.614745 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.614728    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.631233 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.631207    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 07:28:40.631233 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.631214    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 07:28:40.631455 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.631260    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 07:28:40.644258 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.644228    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x4pvw\""
Oct 31 07:28:40.644416 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.644341    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 07:28:40.644599 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.644542    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 07:28:40.644709 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.644610    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 07:28:40.644884 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.644787    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 07:28:40.644884 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.644875    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-oe1fmgg4dv9t\""
Oct 31 07:28:40.645389 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.645374    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 07:28:40.657779 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.657744    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 07:28:40.657779 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.657748    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 07:28:40.657963 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.657772    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 07:28:40.657963 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.657809    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 07:28:40.657963 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.657956    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 07:28:40.658780 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.658759    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 07:28:40.660393 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.660374    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:28:40.663383 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.663360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.663483 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.663397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.663483 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.663428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.663553 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.663490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664049 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5nl\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-kube-api-access-cq5nl\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664118 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664152 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664254 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664237    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664322 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664270    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664362 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664434 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664471 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664518 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664500    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664557 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664546    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664603 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664654 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664640    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664707 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664791 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664854 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664840    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.664906 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.664873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765560 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765560 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765760 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765760 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5nl\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-kube-api-access-cq5nl\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765760 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765760 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765760 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765939 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765939 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765939 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765939 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.765939 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766138 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.765970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766138 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766015    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766138 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766138 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766138 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766419 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766419 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766419 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766578 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766556    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766640 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766604    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.766833 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.766808    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.767250 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.767218    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.768623 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.768604    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.769031 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.769005    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.769495 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.769347    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.769942 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.769640    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.769942 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.769672    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.769942 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.769792    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.770168 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.770139    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.770269 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.770141    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.770359 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.770330    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.770668 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.770642    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.771207 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.771183    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.771431 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.771402    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.771958 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.771939    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.772304 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.772263    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.772776 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.772755    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.786749 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.786724    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5nl\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-kube-api-access-cq5nl\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:40.942153 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:40.942111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:41.135679 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:41.135637    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t" event={"ID":"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea","Type":"ContainerStarted","Data":"e675dc7640c9157bb2a3d1ee7c5e17fb542893f8f230742e3e85771f54d77497"}
Oct 31 07:28:41.136141 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:41.135860    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:41.137423 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:41.137391    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz" event={"ID":"c516d07a-8d53-4cae-9f4e-8b07114916b8","Type":"ContainerStarted","Data":"ea4520f5111b248720a84b3c1c0df7363fd09099ece8b46f17166ce0cd48bbe3"}
Oct 31 07:28:41.140885 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:41.140867    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:41.175350 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:41.175308    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t" podStartSLOduration=1.713368346 podStartE2EDuration="3.175275304s" totalImagesPullingTime="1.461906958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:41.174541165 +0000 UTC m=+55.972225272" watchObservedRunningTime="2026-10-31 07:28:41.175275304 +0000 UTC m=+55.972959444"
Oct 31 07:28:41.253839 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:41.253754    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz" podStartSLOduration=1.9162759230000002 podStartE2EDuration="3.253740389s" totalImagesPullingTime="1.337464466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:41.253412662 +0000 UTC m=+56.051096769" watchObservedRunningTime="2026-10-31 07:28:41.253740389 +0000 UTC m=+56.051424497"
Oct 31 07:28:41.289580 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:41.289547    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 31 07:28:41.292274 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:41.292243    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bf6b3f_71ad_4321_9bbf_4d713e4f2110.slice/crio-8e5f9a7f1b210fbb830f1e68ff7c0dd62636fa7e0ee9ecb050eb680ccd8f99a5 WatchSource:0}: Error finding container 8e5f9a7f1b210fbb830f1e68ff7c0dd62636fa7e0ee9ecb050eb680ccd8f99a5: Status 404 returned error can't find the container with id 8e5f9a7f1b210fbb830f1e68ff7c0dd62636fa7e0ee9ecb050eb680ccd8f99a5
Oct 31 07:28:42.142547 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:42.142512    2577 generic.go:356] "Generic (PLEG): container finished" podID="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" containerID="c16f93a5d8e8d7102bbb53c3e3258b9844b4c26a046a90dfafaff93278cdd944" exitCode=0
Oct 31 07:28:42.142919 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:42.142599    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110","Type":"ContainerDied","Data":"c16f93a5d8e8d7102bbb53c3e3258b9844b4c26a046a90dfafaff93278cdd944"}
Oct 31 07:28:42.142919 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:42.142680    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110","Type":"ContainerStarted","Data":"8e5f9a7f1b210fbb830f1e68ff7c0dd62636fa7e0ee9ecb050eb680ccd8f99a5"}
Oct 31 07:28:43.073447 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:43.073412    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:45.164133 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:45.164099    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110","Type":"ContainerStarted","Data":"b3220b7494d72f4501a5b2f36e9a6500ece78960eb118fdfa8620f757bc4a905"}
Oct 31 07:28:45.164511 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:45.164171    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110","Type":"ContainerStarted","Data":"e367d302195794c7f4778d79e81345ec37614e42bd06c23bade01b5d746bee01"}
Oct 31 07:28:45.164511 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:45.164201    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110","Type":"ContainerStarted","Data":"31db3f844a836393997f68f93ac12d6cff5f7a617fd1c32c5372837f451fe148"}
Oct 31 07:28:45.865577 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:45.865548    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:46.143924 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:46.143899    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:46.170388 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:46.170346    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110","Type":"ContainerStarted","Data":"19dd23a606a4eb6f9a8fd7e2f11e9f07275d36dfe766c23d522814f6e08a22aa"}
Oct 31 07:28:46.170736 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:46.170432    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110","Type":"ContainerStarted","Data":"2e748a061b626b8070f1a84c0b6878c6d6f04600f4156b56229fc99141f9b2f9"}
Oct 31 07:28:46.170736 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:46.170471    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110","Type":"ContainerStarted","Data":"afc725c430ba24a947f07ba62dfb9b7304724847e7c23635855318acfcc15165"}
Oct 31 07:28:46.237584 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:46.237533    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.477506308 podStartE2EDuration="6.237518352s" totalImagesPullingTime="2.745410551s" totalInitContainerRuntime="14.601493ms" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:46.236316313 +0000 UTC m=+61.034000423" watchObservedRunningTime="2026-10-31 07:28:46.237518352 +0000 UTC m=+61.035202459"
Oct 31 07:28:50.561045 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.561000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:50.561486 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.561071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:50.588914 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.588885    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 07:28:50.588914 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.588904    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 07:28:50.594503 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.594475    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:50.594615 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.594482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:50.661908 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.661866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:50.668391 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.668363    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qk9pc\""
Oct 31 07:28:50.674596 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.674574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:50.676123 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.676105    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nzlr\""
Oct 31 07:28:50.676205 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.676189    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 07:28:50.682142 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.682129    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:50.696511 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.696484    2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 07:28:50.704952 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.704924    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:50.942617 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.942588    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:50.973852 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.973824    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-82457"]
Oct 31 07:28:50.976107 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:50.976084    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f4a456_26c2_4a45_bd30_b6795d5c03cc.slice/crio-b8af93f76a20555c304f3ca74d05a669c2d2c3e652cb2b90cd7514f143b58965 WatchSource:0}: Error finding container b8af93f76a20555c304f3ca74d05a669c2d2c3e652cb2b90cd7514f143b58965: Status 404 returned error can't find the container with id b8af93f76a20555c304f3ca74d05a669c2d2c3e652cb2b90cd7514f143b58965
Oct 31 07:28:50.982659 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.982637    2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-548vl\""
Oct 31 07:28:50.988442 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.988419    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s2r9l"]
Oct 31 07:28:50.988529 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:50.988496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:50.990974 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:50.990952    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee91069_0b51_442c_a3c3_71e71302b0e5.slice/crio-26363f8b3ca96c3d4cfa68704cb208679ae1c4f559b96ca316317e75c9fdbe28 WatchSource:0}: Error finding container 26363f8b3ca96c3d4cfa68704cb208679ae1c4f559b96ca316317e75c9fdbe28: Status 404 returned error can't find the container with id 26363f8b3ca96c3d4cfa68704cb208679ae1c4f559b96ca316317e75c9fdbe28
Oct 31 07:28:51.189983 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:51.189949    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-82457" event={"ID":"f0f4a456-26c2-4a45-bd30-b6795d5c03cc","Type":"ContainerStarted","Data":"b8af93f76a20555c304f3ca74d05a669c2d2c3e652cb2b90cd7514f143b58965"}
Oct 31 07:28:51.191010 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:51.190989    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s2r9l" event={"ID":"cee91069-0b51-442c-a3c3-71e71302b0e5","Type":"ContainerStarted","Data":"26363f8b3ca96c3d4cfa68704cb208679ae1c4f559b96ca316317e75c9fdbe28"}
Oct 31 07:28:51.283608 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:51.283585    2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zb9td"]
Oct 31 07:28:51.285518 ip-10-0-140-236 kubenswrapper[2577]: W1031 07:28:51.285489    2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8856b22a_b791_4ca1_8083_650b391429af.slice/crio-c86cae4f41950f6787ed94be7d254929c4cc1502a26a82f8aa50e4af83030c5d WatchSource:0}: Error finding container c86cae4f41950f6787ed94be7d254929c4cc1502a26a82f8aa50e4af83030c5d: Status 404 returned error can't find the container with id c86cae4f41950f6787ed94be7d254929c4cc1502a26a82f8aa50e4af83030c5d
Oct 31 07:28:52.205772 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:52.205736    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zb9td" event={"ID":"8856b22a-b791-4ca1-8083-650b391429af","Type":"ContainerStarted","Data":"c86cae4f41950f6787ed94be7d254929c4cc1502a26a82f8aa50e4af83030c5d"}
Oct 31 07:28:52.209248 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:52.209105    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s2r9l" event={"ID":"cee91069-0b51-442c-a3c3-71e71302b0e5","Type":"ContainerStarted","Data":"34883fbb575f37e34be2a681f96ee214bfb815851eddddd5b2e2bb512d24b01c"}
Oct 31 07:28:53.216344 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:53.216302    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s2r9l" event={"ID":"cee91069-0b51-442c-a3c3-71e71302b0e5","Type":"ContainerStarted","Data":"da1020423e7a57b440bde74f4b7c70f2564dd328eb74155365ac0f0e31b6addd"}
Oct 31 07:28:55.226705 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:55.226669    2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zb9td" event={"ID":"8856b22a-b791-4ca1-8083-650b391429af","Type":"ContainerStarted","Data":"70dea777d1e353a86afc8f2717e9f59c8a6e6ee12fe6c40c95044886c24ebf8a"}
Oct 31 07:28:55.227140 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:55.226815    2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:55.267346 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:55.267263    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zb9td" podStartSLOduration=67.216752938 podStartE2EDuration="1m10.26724187s" totalImagesPullingTime="3.050488932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:55.266174918 +0000 UTC m=+70.063859027" watchObservedRunningTime="2026-10-31 07:28:55.26724187 +0000 UTC m=+70.064925980"
Oct 31 07:28:55.267917 ip-10-0-140-236 kubenswrapper[2577]: I1031 07:28:55.267867    2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s2r9l" podStartSLOduration=69.197322173 podStartE2EDuration="1m10.267851666s" totalImagesPullingTime="1.070529493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:53.257855553 +0000 UTC m=+68.055539661" watchObservedRunningTime="2026-10-31 07:28:55.267851666 +0000 UTC m=+70.065535775"
Oct 31 07:28:55.781428 ip-10-0-140-236 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 07:28:55.797625 ip-10-0-140-236 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 07:28:55.797841 ip-10-0-140-236 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 07:28:55.797922 ip-10-0-140-236 systemd[1]: kubelet.service: Consumed 4.487s CPU time.
Oct 31 07:28:55.810805 ip-10-0-140-236 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 07:28:55.891300 ip-10-0-140-236 kubenswrapper[5829]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:55.891300 ip-10-0-140-236 kubenswrapper[5829]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 07:28:55.891300 ip-10-0-140-236 kubenswrapper[5829]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:55.891300 ip-10-0-140-236 kubenswrapper[5829]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 07:28:55.893406 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893322    5829 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:55.893406 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893405    5829 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:55.893406 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893409    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893413    5829 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893416    5829 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893419    5829 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893421    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893424    5829 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893427    5829 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893429    5829 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893432    5829 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893435    5829 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893437    5829 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893439    5829 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893442    5829 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893444    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893447    5829 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893450    5829 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893453    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893455    5829 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893458    5829 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893460    5829 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:55.893497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893463    5829 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893466    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893468    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893471    5829 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893473    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893476    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893479    5829 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893481    5829 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893484    5829 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893487    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893490    5829 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893492    5829 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893495    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893501    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893503    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893506    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893509    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893512    5829 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893514    5829 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:55.893959 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893517    5829 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893520    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893523    5829 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893525    5829 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893528    5829 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893531    5829 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893533    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893536    5829 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893538    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893541    5829 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893545    5829 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893549    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893552    5829 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893555    5829 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893557    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893560    5829 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893562    5829 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893565    5829 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893568    5829 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:55.894424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893571    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893573    5829 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893575    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893578    5829 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893581    5829 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893583    5829 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893586    5829 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893591    5829 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893595    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893598    5829 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893600    5829 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893603    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893605    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893608    5829 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893611    5829 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893613    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893616    5829 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893618    5829 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893621    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893624    5829 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:55.894855 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893626    5829 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893629    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893631    5829 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893634    5829 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893636    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893639    5829 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893642    5829 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893644    5829 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893647    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893649    5829 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893652    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893654    5829 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893657    5829 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893660    5829 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893663    5829 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893665    5829 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893668    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893670    5829 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893673    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893677    5829 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:55.895340 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893680    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893682    5829 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893685    5829 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893687    5829 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893690    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.893695    5829 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894490    5829 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894497    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894500    5829 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894503    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894506    5829 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894509    5829 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894512    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894515    5829 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894517    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894520    5829 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894523    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894525    5829 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894528    5829 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:55.895816 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894531    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894534    5829 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894538    5829 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894542    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894545    5829 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894548    5829 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894554    5829 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894556    5829 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894559    5829 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894562    5829 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894564    5829 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894568    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894571    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894574    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894577    5829 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894579    5829 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894582    5829 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894585    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:55.896272 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894587    5829 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894591    5829 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894595    5829 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894598    5829 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894601    5829 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894603    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894606    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894609    5829 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894611    5829 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894613    5829 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894616    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894618    5829 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894621    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894623    5829 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894626    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894628    5829 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894631    5829 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894633    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894636    5829 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:55.896752 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894639    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894641    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894645    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894648    5829 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894650    5829 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894653    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894656    5829 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894658    5829 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894661    5829 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894664    5829 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894666    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894669    5829 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894672    5829 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894674    5829 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894677    5829 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894679    5829 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894682    5829 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894684    5829 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894687    5829 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894689    5829 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:55.897193 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894692    5829 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894695    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894697    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894700    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894702    5829 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894705    5829 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894707    5829 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894710    5829 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894713    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894715    5829 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894718    5829 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894720    5829 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894723    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894725    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894728    5829 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894731    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894733    5829 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894736    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894739    5829 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894741    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:55.897671 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894744    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894746    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894749    5829 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894752    5829 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894754    5829 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894757    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894759    5829 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894762    5829 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894765    5829 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894767    5829 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894770    5829 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894773    5829 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894775    5829 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894777    5829 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894780    5829 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.894783    5829 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895300    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895307    5829 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895310    5829 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895313    5829 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:55.898170 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895316    5829 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895318    5829 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895321    5829 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895324    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895326    5829 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895329    5829 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895332    5829 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895335    5829 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895338    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895340    5829 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895344    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895347    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895349    5829 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895352    5829 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895355    5829 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895358    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895360    5829 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895363    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895365    5829 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895368    5829 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:55.898652 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895371    5829 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895373    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895376    5829 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895380    5829 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895384    5829 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895387    5829 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895390    5829 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895392    5829 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895395    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895398    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895400    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895403    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895406    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895409    5829 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895412    5829 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895414    5829 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895417    5829 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895420    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895422    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:55.899144 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895424    5829 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895427    5829 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895433    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895437    5829 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895441    5829 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895444    5829 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895447    5829 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895450    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895452    5829 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895455    5829 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895458    5829 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895461    5829 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895463    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895466    5829 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895469    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895471    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895473    5829 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895476    5829 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895479    5829 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:55.899633 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895481    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895484    5829 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895486    5829 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895489    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895491    5829 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895494    5829 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895496    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895499    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895501    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895504    5829 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895506    5829 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895509    5829 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895511    5829 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895514    5829 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895516    5829 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895527    5829 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895530    5829 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895532    5829 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895535    5829 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:55.900085 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895537    5829 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895540    5829 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895542    5829 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895545    5829 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895547    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895550    5829 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895553    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895556    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895558    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895561    5829 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895563    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895567    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895569    5829 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895572    5829 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895575    5829 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895577    5829 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895580    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895582    5829 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895584    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895587    5829 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:55.900544 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895589    5829 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895592    5829 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895594    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895597    5829 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.895599    5829 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895691    5829 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895698    5829 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895711    5829 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895717    5829 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895730    5829 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895734    5829 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895738    5829 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895743    5829 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895746    5829 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895749    5829 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895753    5829 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895756    5829 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895759    5829 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895762    5829 flags.go:64] FLAG: --cgroup-root=""
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895765    5829 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895768    5829 flags.go:64] FLAG: --client-ca-file=""
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895771    5829 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895774    5829 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 07:28:55.901005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895781    5829 flags.go:64] FLAG: --cluster-domain=""
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895784    5829 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895788    5829 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895791    5829 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895794    5829 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895798    5829 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895801    5829 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895804    5829 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895808    5829 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895811    5829 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895814    5829 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895817    5829 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895820    5829 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895823    5829 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895827    5829 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895831    5829 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895833    5829 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895837    5829 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895841    5829 flags.go:64] FLAG: --enable-server="true"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895849    5829 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895860    5829 flags.go:64] FLAG: --event-burst="100"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895863    5829 flags.go:64] FLAG: --event-qps="50"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895866    5829 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895869    5829 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 07:28:55.901551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895872    5829 flags.go:64] FLAG: --eviction-hard=""
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895876    5829 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895879    5829 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895882    5829 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895885    5829 flags.go:64] FLAG: --eviction-soft=""
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895888    5829 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895891    5829 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895894    5829 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895897    5829 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895900    5829 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895903    5829 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895906    5829 flags.go:64] FLAG: --feature-gates=""
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895919    5829 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895923    5829 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895926    5829 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895929    5829 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895932    5829 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895935    5829 flags.go:64] FLAG: --help="false"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895938    5829 flags.go:64] FLAG: --hostname-override="ip-10-0-140-236.ec2.internal"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895941    5829 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895945    5829 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895947    5829 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895951    5829 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 07:28:55.902133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895954    5829 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895957    5829 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895963    5829 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895966    5829 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895969    5829 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895978    5829 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895987    5829 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895990    5829 flags.go:64] FLAG: --kube-reserved=""
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895993    5829 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895996    5829 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.895999    5829 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896002    5829 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896005    5829 flags.go:64] FLAG: --lock-file=""
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896007    5829 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896010    5829 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896014    5829 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896019    5829 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896022    5829 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896025    5829 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896030    5829 flags.go:64] FLAG: --logging-format="text"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896033    5829 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896037    5829 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896039    5829 flags.go:64] FLAG: --manifest-url=""
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896042    5829 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896047    5829 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 07:28:55.902698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896050    5829 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896054    5829 flags.go:64] FLAG: --max-pods="110"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896057    5829 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896060    5829 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896063    5829 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896065    5829 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896068    5829 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896071    5829 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896074    5829 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896082    5829 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896085    5829 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896088    5829 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896092    5829 flags.go:64] FLAG: --pod-cidr=""
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896100    5829 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896103    5829 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896106    5829 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896109    5829 flags.go:64] FLAG: --port="10250"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896112    5829 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896115    5829 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c969980dfd1b6c16"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896118    5829 flags.go:64] FLAG: --qos-reserved=""
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896121    5829 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896124    5829 flags.go:64] FLAG: --register-node="true"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896127    5829 flags.go:64] FLAG: --register-with-taints=""
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896131    5829 flags.go:64] FLAG: --registry-burst="10"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896134    5829 flags.go:64] FLAG: --registry-qps="5"
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896137    5829 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 07:28:55.903331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896141    5829 flags.go:64] FLAG: --reserved-memory=""
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896145    5829 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896154    5829 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896157    5829 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896160    5829 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896163    5829 flags.go:64] FLAG: --runonce="false"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896166    5829 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896169    5829 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896172    5829 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896175    5829 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896178    5829 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896181    5829 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896184    5829 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896187    5829 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896190    5829 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896194    5829 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896197    5829 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896200    5829 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896203    5829 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896206    5829 flags.go:64] FLAG: --system-cgroups=""
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896215    5829 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896221    5829 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896224    5829 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896227    5829 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896234    5829 flags.go:64] FLAG: --tls-min-version=""
Oct 31 07:28:55.903940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896237    5829 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896240    5829 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896243    5829 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896246    5829 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896249    5829 flags.go:64] FLAG: --v="2"
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896253    5829 flags.go:64] FLAG: --version="false"
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896257    5829 flags.go:64] FLAG: --vmodule=""
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896262    5829 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896266    5829 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896456    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896461    5829 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896464    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896467    5829 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896470    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896472    5829 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896475    5829 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896477    5829 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896480    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896482    5829 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896485    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896488    5829 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:55.904521 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896490    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896494    5829 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896497    5829 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896501    5829 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896504    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896508    5829 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896511    5829 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896519    5829 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896522    5829 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896525    5829 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896527    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896530    5829 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896533    5829 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896535    5829 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896538    5829 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896541    5829 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896543    5829 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896546    5829 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896550    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896553    5829 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:55.905328 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896556    5829 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896558    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896561    5829 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896563    5829 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896566    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896568    5829 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896571    5829 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896573    5829 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896576    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896578    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896581    5829 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896583    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896586    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896590    5829 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896593    5829 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896596    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896598    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896601    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896603    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896606    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:55.906091 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896614    5829 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896617    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896620    5829 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896622    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896625    5829 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896628    5829 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896630    5829 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896633    5829 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896636    5829 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896638    5829 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896642    5829 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896645    5829 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896648    5829 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896650    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896653    5829 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896655    5829 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896658    5829 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896660    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896663    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:55.906747 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896665    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896668    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896671    5829 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896673    5829 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896677    5829 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896681    5829 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896686    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896688    5829 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896691    5829 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896694    5829 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896696    5829 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896699    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896702    5829 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896705    5829 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896713    5829 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896715    5829 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896718    5829 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896720    5829 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896723    5829 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:55.907206 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896726    5829 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896728    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896731    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896733    5829 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896737    5829 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896740    5829 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896742    5829 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896745    5829 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896748    5829 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896751    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896753    5829 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896756    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896759    5829 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896762    5829 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896764    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896767    5829 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:55.907885 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.896793    5829 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896834    5829 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896838    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896842    5829 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896846    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896849    5829 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896851    5829 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896854    5829 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896857    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896859    5829 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896862    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896865    5829 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896875    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896878    5829 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896880    5829 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896884    5829 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896888    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896891    5829 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896894    5829 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:55.908367 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896897    5829 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896901    5829 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896904    5829 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896907    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896910    5829 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896912    5829 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896915    5829 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896917    5829 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896920    5829 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896922    5829 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896925    5829 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896927    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896930    5829 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896932    5829 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896935    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896938    5829 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896940    5829 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896944    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896946    5829 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896949    5829 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:55.908910 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896952    5829 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896954    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896957    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896959    5829 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896962    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896964    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896972    5829 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896975    5829 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896978    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896981    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896983    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896985    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896988    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896993    5829 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896995    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.896998    5829 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897001    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897003    5829 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897006    5829 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:55.909517 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897008    5829 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897011    5829 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897014    5829 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897016    5829 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897019    5829 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897021    5829 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897024    5829 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897026    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897029    5829 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897031    5829 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897035    5829 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897038    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897040    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897043    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897045    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897048    5829 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897050    5829 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897055    5829 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897057    5829 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:55.909966 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897060    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897068    5829 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897071    5829 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897074    5829 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897076    5829 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897079    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897082    5829 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897086    5829 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897088    5829 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897091    5829 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897093    5829 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897096    5829 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897098    5829 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897101    5829 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897103    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897106    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897109    5829 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897111    5829 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897114    5829 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897116    5829 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:55.910575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897119    5829 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897122    5829 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897124    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897128    5829 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897131    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897133    5829 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897136    5829 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897138    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897141    5829 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.897144    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.901476    5829 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.901492    5829 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.901511    5829 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.901516    5829 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901595    5829 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901608    5829 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901611    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901614    5829 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901617    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:55.911194 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901620    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901624    5829 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901628    5829 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901631    5829 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901641    5829 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901645    5829 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901648    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901651    5829 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901653    5829 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901656    5829 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901659    5829 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901661    5829 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901664    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901667    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901670    5829 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901672    5829 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901675    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901678    5829 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901680    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:55.911939 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901683    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901685    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901688    5829 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901691    5829 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901693    5829 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901696    5829 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901698    5829 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901701    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901705    5829 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901708    5829 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901710    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901713    5829 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901715    5829 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901718    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901721    5829 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901723    5829 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901726    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901728    5829 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901737    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:55.912424 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901740    5829 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901742    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901745    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901747    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901750    5829 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901753    5829 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901755    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901758    5829 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901761    5829 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901763    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901765    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901768    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901772    5829 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901774    5829 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901777    5829 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901780    5829 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901782    5829 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901785    5829 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901787    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901790    5829 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:55.912866 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901792    5829 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901795    5829 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901797    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901800    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901802    5829 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901805    5829 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901808    5829 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901810    5829 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901813    5829 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901815    5829 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901818    5829 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901820    5829 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901828    5829 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901831    5829 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901834    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901836    5829 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901839    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901841    5829 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901844    5829 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901847    5829 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:55.913501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901850    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901852    5829 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901855    5829 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901858    5829 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901860    5829 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901863    5829 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901867    5829 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901870    5829 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901873    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901876    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901879    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901881    5829 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901884    5829 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901886    5829 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901889    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901892    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901894    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901897    5829 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901899    5829 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:55.914001 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901902    5829 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901904    5829 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901907    5829 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.901909    5829 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.901937    5829 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902164    5829 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902175    5829 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902178    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902181    5829 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902183    5829 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902187    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902190    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902193    5829 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902195    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902198    5829 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902200    5829 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:55.914460 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902203    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902206    5829 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902208    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902211    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902213    5829 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902216    5829 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902219    5829 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902222    5829 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902226    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902229    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902232    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902245    5829 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902248    5829 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902250    5829 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902253    5829 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902256    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902258    5829 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902261    5829 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902265    5829 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:55.914840 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902268    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902271    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902274    5829 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902276    5829 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902300    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902303    5829 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902306    5829 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902310    5829 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902312    5829 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902315    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902318    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902321    5829 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902323    5829 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902326    5829 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902329    5829 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902332    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902334    5829 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902337    5829 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902339    5829 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:55.915393 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902342    5829 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902344    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902347    5829 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902350    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902352    5829 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902355    5829 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902357    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902359    5829 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902362    5829 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902364    5829 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902367    5829 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902370    5829 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902372    5829 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902375    5829 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902377    5829 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902380    5829 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902383    5829 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902385    5829 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902393    5829 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902396    5829 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:55.915972 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902399    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902401    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902404    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902407    5829 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902409    5829 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902411    5829 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902414    5829 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902417    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902419    5829 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902422    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902424    5829 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902427    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902430    5829 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902432    5829 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902435    5829 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902437    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902439    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902442    5829 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902444    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:55.916640 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902447    5829 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902449    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902452    5829 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902454    5829 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902457    5829 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902459    5829 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902462    5829 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902464    5829 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902466    5829 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902469    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902472    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902474    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902484    5829 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902486    5829 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902489    5829 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902492    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902494    5829 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:55.917249 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902497    5829 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.902520    5829 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902570    5829 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902574    5829 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902577    5829 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902581    5829 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902584    5829 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902587    5829 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902590    5829 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902592    5829 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902595    5829 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902597    5829 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902600    5829 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902603    5829 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902613    5829 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902616    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 07:28:55.917925 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902619    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902621    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902624    5829 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902627    5829 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902629    5829 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902632    5829 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902634    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902637    5829 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902640    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902642    5829 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902645    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902647    5829 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902655    5829 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902658    5829 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902661    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902663    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902666    5829 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902668    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902671    5829 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 07:28:55.918401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902677    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902680    5829 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902682    5829 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902685    5829 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902688    5829 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902690    5829 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902693    5829 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902695    5829 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902698    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902700    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902703    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902706    5829 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902708    5829 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902711    5829 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902714    5829 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902716    5829 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902719    5829 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902722    5829 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902725    5829 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902727    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 07:28:55.918852 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902730    5829 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902732    5829 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902736    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902738    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902741    5829 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902743    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902751    5829 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902754    5829 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902757    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902759    5829 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902762    5829 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902764    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902768    5829 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902771    5829 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902773    5829 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902777    5829 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902781    5829 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902784    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902787    5829 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 07:28:55.919373 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902790    5829 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902792    5829 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902795    5829 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902798    5829 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902800    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902803    5829 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902806    5829 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902810    5829 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902813    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902817    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902820    5829 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902823    5829 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902825    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902828    5829 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902830    5829 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902833    5829 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902836    5829 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902839    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902841    5829 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 07:28:55.919818 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902844    5829 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902853    5829 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902856    5829 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902859    5829 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902861    5829 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902865    5829 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902869    5829 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902871    5829 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902874    5829 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902876    5829 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902879    5829 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902882    5829 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902884    5829 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902887    5829 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:28:55.902890    5829 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.902951    5829 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.903977    5829 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.904036    5829 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.904243    5829 server.go:1014] "Starting client certificate rotation"
Oct 31 07:28:55.920255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.904453    5829 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 07:28:55.920760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.904529    5829 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:22:45 +0000 UTC" deadline="2028-06-11 17:57:06.33288982 +0000 UTC"
Oct 31 07:28:55.920760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.904569    5829 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14146h28m10.428322404s"
Oct 31 07:28:55.920760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.905584    5829 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:28:55.920760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.907389    5829 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 07:28:55.920760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.908671    5829 log.go:25] "Validated CRI v1 runtime API"
Oct 31 07:28:55.920760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.910758    5829 log.go:25] "Validated CRI v1 image API"
Oct 31 07:28:55.920760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.911560    5829 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 07:28:55.920760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.913926    5829 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d173a70e-bbe9-4b8c-bf85-09694211657f:/dev/nvme0n1p4 f82f9d83-cdba-4c31-b103-4aceb0aaae16:/dev/nvme0n1p3]
Oct 31 07:28:55.920992 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.913945    5829 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c50ba8c8f775968ea83b50a1cffe3e94b895e7dd2c0f6d7da6fc47edcb573d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c50ba8c8f775968ea83b50a1cffe3e94b895e7dd2c0f6d7da6fc47edcb573d2/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26363f8b3ca96c3d4cfa68704cb208679ae1c4f559b96ca316317e75c9fdbe28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26363f8b3ca96c3d4cfa68704cb208679ae1c4f559b96ca316317e75c9fdbe28/userdata/shm major:0 minor:477 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b922b9e6f40c1835b16f980b82d23e256f7f2527e59a0553cd0804c32f83ea8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b922b9e6f40c1835b16f980b82d23e256f7f2527e59a0553cd0804c32f83ea8/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4135f17c15b08ddff40838d0874e901eb5560c4d09c07d2e62eb661c88c198cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4135f17c15b08ddff40838d0874e901eb5560c4d09c07d2e62eb661c88c198cb/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70f074b267a802533fc01318aebb2b926b621bfa14a701947086928a0c04e779/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70f074b267a802533fc01318aebb2b926b621bfa14a701947086928a0c04e779/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/826e797743f1a8ab0415c3d272c61f0319adfa5509a314b51673345fe47f394d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/826e797743f1a8ab0415c3d272c61f0319adfa5509a314b51673345fe47f394d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8401c67bdc047a50ee8712abf8fc395becfaa60b763a3985fab0d4d539031cc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8401c67bdc047a50ee8712abf8fc395becfaa60b763a3985fab0d4d539031cc9/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a1106ff94c3a8aa7b24c8e86d66006d4fa4e9d6ac46c1f682b5eee6af2ed14e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a1106ff94c3a8aa7b24c8e86d66006d4fa4e9d6ac46c1f682b5eee6af2ed14e/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e5f9a7f1b210fbb830f1e68ff7c0dd62636fa7e0ee9ecb050eb680ccd8f99a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e5f9a7f1b210fbb830f1e68ff7c0dd62636fa7e0ee9ecb050eb680ccd8f99a5/userdata/shm major:0 minor:438 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/943a503a9bc0ee6a374d29505f320454a217981af0d72d26163429f181391af3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/943a503a9bc0ee6a374d29505f320454a217981af0d72d26163429f181391af3/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b10279343c85685543df2671b87124ea3283151b3c3ce9771b68f6fc1bbdba49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b10279343c85685543df2671b87124ea3283151b3c3ce9771b68f6fc1bbdba49/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b72f45ccc31af5137314f6e552b5db1ff3eae84c2e41fcc5cd2b38593ebb6cee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b72f45ccc31af5137314f6e552b5db1ff3eae84c2e41fcc5cd2b38593ebb6cee/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8af93f76a20555c304f3ca74d05a669c2d2c3e652cb2b90cd7514f143b58965/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8af93f76a20555c304f3ca74d05a669c2d2c3e652cb2b90cd7514f143b58965/userdata/shm major:0 minor:475 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba447568733a33979f88bcdce24ef1ace1d241fe20e8b32a37135e2d1ba9a0c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba447568733a33979f88bcdce24ef1ace1d241fe20e8b32a37135e2d1ba9a0c9/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c24134f727811f6856dab38b787722771adb75015ff707ed8b73e7e7ecfe53f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c24134f727811f6856dab38b787722771adb75015ff707ed8b73e7e7ecfe53f2/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c86cae4f41950f6787ed94be7d254929c4cc1502a26a82f8aa50e4af83030c5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c86cae4f41950f6787ed94be7d254929c4cc1502a26a82f8aa50e4af83030c5d/userdata/shm major:0 minor:484 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cce489912507a7128eabc59d8e95f8d1145bf3fb49f84ca8f2639aefba65e84b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cce489912507a7128eabc59d8e95f8d1145bf3fb49f84ca8f2639aefba65e84b/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3667fbda52b872511daaaf50ba7c878c6daadd2b38b2969636c77e8673b2e02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3667fbda52b872511daaaf50ba7c878c6daadd2b38b2969636c77e8673b2e02/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da6208239c5f55b66611f1e47bb739d135e8abef821c0c42f3919c7c468f978e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da6208239c5f55b66611f1e47bb739d135e8abef821c0c42f3919c7c468f978e/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db53f4ce4263659743fb99007a507a0a5ee3401bf56259ba3de1b22515d21de0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db53f4ce4263659743fb99007a507a0a5ee3401bf56259ba3de1b22515d21de0/userdata/shm major:0 minor:378 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dcc41609851bea2369d498de074702337e8bdda82cb0dff5822362ffc4d85974/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dcc41609851bea2369d498de074702337e8bdda82cb0dff5822362ffc4d85974/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e23e9f7f6762228112a7c326718a42286611483fda8838fa797b01cbf9c19090/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e23e9f7f6762228112a7c326718a42286611483fda8838fa797b01cbf9c19090/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e9d25f1c3282d0520135486e3cfbe329989f8d3152940e6988a10910b6e59f4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e9d25f1c3282d0520135486e3cfbe329989f8d3152940e6988a10910b6e59f4c/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed8c10ee633e1ad18b4096e6c91ded20f77169bd3a89deab0b0c90b981179749/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed8c10ee633e1ad18b4096e6c91ded20f77169bd3a89deab0b0c90b981179749/userdata/shm major:0 minor:342 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5/volumes/kubernetes.io~projected/kube-api-access-6tr9c:{mountpoint:/var/lib/kubelet/pods/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5/volumes/kubernetes.io~projected/kube-api-access-6tr9c major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/132e4deb-32cc-4b7b-a179-44a9700f1bac/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/132e4deb-32cc-4b7b-a179-44a9700f1bac/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e3a7ead-4527-476b-9b02-c50e68e80166/volumes/kubernetes.io~projected/kube-api-access-cx78k:{mountpoint:/var/lib/kubelet/pods/1e3a7ead-4527-476b-9b02-c50e68e80166/volumes/kubernetes.io~projected/kube-api-access-cx78k major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2af799e3-eb79-41f6-be34-411ea2f10326/volumes/kubernetes.io~projected/kube-api-access-cvhrd:{mountpoint:/var/lib/kubelet/pods/2af799e3-eb79-41f6-be34-411ea2f10326/volumes/kubernetes.io~projected/kube-api-access-cvhrd major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/volumes/kubernetes.io~projected/kube-api-access-cffpp:{mountpoint:/var/lib/kubelet/pods/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/volumes/kubernetes.io~projected/kube-api-access-cffpp major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/457dde04-3e55-4c30-a1c5-d13672f48a4a/volumes/kubernetes.io~projected/kube-api-access-bcflp:{mountpoint:/var/lib/kubelet/pods/457dde04-3e55-4c30-a1c5-d13672f48a4a/volumes/kubernetes.io~projected/kube-api-access-bcflp major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/457dde04-3e55-4c30-a1c5-d13672f48a4a/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/457dde04-3e55-4c30-a1c5-d13672f48a4a/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/volumes/kubernetes.io~projected/kube-api-access-m7b5w:{mountpoint:/var/lib/kubelet/pods/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/volumes/kubernetes.io~projected/kube-api-access-m7b5w major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fc187c9-2163-4de8-afbb-77f65de777c8/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4fc187c9-2163-4de8-afbb-77f65de777c8/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fc187c9-2163-4de8-afbb-77f65de777c8/volumes/kubernetes.io~projected/kube-api-access-4r8cv:{mountpoint:/var/lib/kubelet/pods/4fc187c9-2163-4de8-afbb-77f65de777c8/volumes/kubernetes.io~projected/kube-api-access-4r8cv major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fc187c9-2163-4de8-afbb-77f65de777c8/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4fc187c9-2163-4de8-afbb-77f65de777c8/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/658d33ef-d90b-448b-b4d0-e175b77a187f/volumes/kubernetes.io~projected/kube-api-access-mpnv5:{mountpoint:/var/lib/kubelet/pods/658d33ef-d90b-448b-b4d0-e175b77a187f/volumes/kubernetes.io~projected/kube-api-access-mpnv5 major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/658d33ef-d90b-448b-b4d0-e175b77a187f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/658d33ef-d90b-448b-b4d0-e175b77a187f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/658d33ef-d90b-448b-b4d0-e175b77a187f/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/658d33ef-d90b-448b-b4d0-e175b77a187f/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836cf52b-b0e8-4e98-a812-21d0cebfaf63/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/836cf52b-b0e8-4e98-a812-21d0cebfaf63/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836cf52b-b0e8-4e98-a812-21d0cebfaf63/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/836cf52b-b0e8-4e98-a812-21d0cebfaf63/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/836cf52b-b0e8-4e98-a812-21d0cebfaf63/volumes/kubernetes.io~projected/kube-api-access-66fgg:{mountpoint:/var/lib/kubelet/pods/836cf52b-b0e8-4e98-a812-21d0cebfaf63/volumes/kubernetes.io~projected/kube-api-access-66fgg major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8856b22a-b791-4ca1-8083-650b391429af/volumes/kubernetes.io~projected/kube-api-access-h4944:{mountpoint:/var/lib/kubelet/pods/8856b22a-b791-4ca1-8083-650b391429af/volumes/kubernetes.io~projected/kube-api-access-h4944 major:0 minor:479 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95b1f652-e473-4b0e-a9da-ac32940f95a8/volumes/kubernetes.io~projected/kube-api-access-kxz6l:{mountpoint:/var/lib/kubelet/pods/95b1f652-e473-4b0e-a9da-ac32940f95a8/volumes/kubernetes.io~projected/kube-api-access-kxz6l major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95b1f652-e473-4b0e-a9da-ac32940f95a8/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/95b1f652-e473-4b0e-a9da-ac32940f95a8/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/95b1f652-e473-4b0e-a9da-ac32940f95a8/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/95b1f652-e473-4b0e-a9da-ac32940f95a8/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7/volumes/kubernetes.io~projected/kube-api-access-nsnkc:{mountpoint:/var/lib/kubelet/pods/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7/volumes/kubernetes.io~projected/kube-api-access-nsnkc major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b62296f1-d84a-4dd1-8eb0-9f64567af021/volumes/kubernetes.io~projected/kube-api-access-52lps:{mountpoint:/var/lib/kubelet/pods/b62296f1-d84a-4dd1-8eb0-9f64567af021/volumes/kubernetes.io~projected/kube-api-access-52lps major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b62296f1-d84a-4dd1-8eb0-9f64567af021/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b62296f1-d84a-4dd1-8eb0-9f64567af021/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/web-config/alertmanager/9 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/web-config/config-reloader/6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~empty-dir/config-out major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~projected/kube-api-access-x2wzb:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~projected/kube-api-access-x2wzb major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~projected/tls-assets major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/config-volume major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/web-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volume-subpaths/web-config/config-reloader/2 major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volume-subpaths/web-config/prometheus/15 major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~empty-dir/config-out major:0 minor:422 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~projected/kube-api-access-cq5nl:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~projected/kube-api-access-cq5nl major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~projected/tls-assets major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/config major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:429 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:430 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/web-config major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~projected/kube-api-access-qfg4b:{mountpoint:/var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~projected/kube-api-access-qfg4b major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cee91069-0b51-442c-a3c3-71e71302b0e5/volumes/kubernetes.io~projected/kube-api-access-lkpsk:{mountpoint:/var/lib/kubelet/pods/cee91069-0b51-442c-a3c3-71e71302b0e5/volumes/kubernetes.io~projected/kube-api-access-lkpsk major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cee91069-0b51-442c-a3c3-71e71302b0e5/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/cee91069-0b51-442c-a3c3-71e71302b0e5/volumes/kubernetes.io~secret/metrics-certs major:0 minor:470 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db612775-57b2-405c-a403-8c78f173d018/volumes/kubernetes.io~projected/kube-api-access-g7vps:{mountpoint:/var/lib/kubelet/pods/db612775-57b2-405c-a403-8c78f173d018/volumes/kubernetes.io~projected/kube-api-access-g7vps major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea/volumes/kubernetes.io~projected/kube-api-access-w6l87:{mountpoint:/var/lib/kubelet/pods/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea/volumes/kubernetes.io~projected/kube-api-access-w6l87 major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f4a456-26c2-4a45-bd30-b6795d5c03cc/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f0f4a456-26c2-4a45-bd30-b6795d5c03cc/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f206a32f-819d-4e61-9040-790e8c9183ec/volumes/kubernetes.io~projected/kube-api-access-xbx2j:{mountpoint:/var/lib/kubelet/pods/f206a32f-819d-4e61-9040-790e8c9183ec/volumes/kubernetes.io~projected/kube-api-access-xbx2j major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~projected/kube-api-access-hgglw:{mountpoint:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~projected/kube-api-access-hgglw major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:319 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/4d86dbdbd0fb2e27265698247f18ac05d595419f4b9941ec70cf6b5f963f453c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2bea1ae72e14153afadcee84f7b85ce5c1d69b0babd1c2af4834deaa4da2f062/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/01f085c44bc2a9239ece1083c925454eb22b79018606729791a139cf5a827df6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/9b3eb84641c475149bd8dde676f1d914c6914cb391163407e3d1d2ac870b3271/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3e966f4a2040dbc25202c818bade7062657724a2a5567cdc5ce70de535f3eba8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b47edfad714c6266779f1832922ee36159529b303ecb7a0e9fa6d517a7a2fd8c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5196cee1da9c07f34f7eb102dcdac565ebb0c669ca6f4cc04e65124a6ffe519d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ee9f89e48227ca3fe6cc0f31cb712bb373688654c6e9fb48870aa21111cfa97a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/4b8b7ef164c010ce07e3b684175beb9e2164d7d66511678668e143e6e7a6d382/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/dfa9561c00b64e1817d4a23186297544be4fa0bf1d827e483c2664be2b4a5241/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/4b93ec8d3af671f7a04176b6ea4b5a663815ae898b3ed182d816cd52db0bd91c/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/bf71425e12af4cf2d21a2aaa61579493fb65982d91195a73b8cf528565b98b60/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1c5ec5d3f3ab1e5d6eedf46648244c686cbd4664ad8be311d95a995b8e112761/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d2927318af95ef429c06bd78448932f2a437b8ea2dff3bc9b7f7ad5a410fb105/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/353f679a5e04097ef2cf7a5fe5fde967329b6954be3784677527e96253967d65/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cb345a81961ecce9237ae26bfc8811d5622d958db9280c19691d1b67b6d535a1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2a34bfb74768ac21579e5e36c58c128a9d7956a18b5bc67b1a76de7fd4fd57e2/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/dec0505e56f6782050132a304d93bf3fabd4080688ce00ef3d1eab57ba4047c0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/46989b4b09448b55060f47a44ed122d1dc62538b3d74a748106e04bec57c6239/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/45261ca4bad9fcf880bb9085874e8e31c2e26a066f16ff0443653e869af9ab68/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/53a78eb76e10cd6072c110ee75b6eb59696a1e07245552ebf7f5708b755433ff/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/fb9366ad50f2217b61eeb2cdabf22f58f6c8ac41a278d77434b8d5878a42623d/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/ce8684e1f8b5efbc2671f2376f8707cfccead589cdc3c0527ff2671aebffd486/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/c0938bbe3bcb22a16b829eae347bc441133bf72ed34d0420294cb42dd7d5738a/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/51c8aba25db79abf077f40449299fd1d06cbc601cebc7a13002972385771bd88/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/8e8d368b6014a4f75548af4524ae38f191f4530851e41bac1591cab10735f864/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/6648068230bf84c9ed6cfa6d65cf0d5aaf2183a2a79a8c410a221190cd2e00bc/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/d8bd8d65fb0f34e3819a0ae658add7b191a6f0250323e22c2e2f49df6602f1db/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/9452a5d3a7876710cac965c9b3149a3bf70a307f3b13d0d891fb21e859f1e6e1/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/353f2e8bafd5143236b30549c822c57133aab6c18130d00df780a55b40d70bb9/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/769add60235b87fd9c44acba49fba26e859a58c067c2096159e921076ac953c6/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/a7a5959002ff1b69754cd9612884a28c38e045b70308794b6ed470b7b3d32623/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/8649c3657e84e6da0f8f8a3b426b2a3f2f0b04bce4b996f9587d3e339b49c382/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/abea56db03e444eed31b887efedd556eac6a8af898b37f214d64b61de9014779/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/eb69d5d521c846dd5e5636927ff6ca6bb1084bf359efd4248cf723070c236d16/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/45300dd75bd374b04749825a96f03ad9ecc19296d1bcd13ef47ac59fdd5a7cd0/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/9eb857dcdaec522be65d8f1381cffc77b345c0b1f9a5e079049ffde30976572d/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/37148d0a1396262be237304ca1cc01bc34d31310bb6c2dfeac61848d4ba379e8/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/7ea277d9a70b994deaed90ce6a9c212307473f88f85510de714d700ea6f09b7c/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/7bb08808adf0c99de2719aadb4936cac13e0fbfcf863138b8ff29b22f24b61a5/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/57dce49c7106fcd5e8b4196e2620f87c7ef928f0233d3c798849d7748762a076/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/fff8aaf5a149dcca5a5464ea924e76e14780af09a83563a624a23ff933b5eaf5/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/64a19d08521191d66d5780d53e4f2be1e23030a7a29816e0fbf1963bd794b3b9/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/92f74470be25d99f5c558aa4ca6fc83e38c3fe711c88495b0c6a415d37ee29ce/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/c27b8b8a1e506015698e5b0ff8118e05a6a7849b9435a1395e6f2b32773419c6/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/a28754fab33dd6507968ca7cb2c60079fb844372c5bebd80cc7648af0820371e/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/d5405926cbd8c44935ba362d3721f3e903c94647c15f533c5ffc1c8249b10c57/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/da0fa57725f11abf71830f2ec4b83a01d7716d2e572159254fc1ffeb1a48af5e/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/f80aa2fa42da0e270007d226be3d7c107158a0fcb595c86a211f85b3950eb566/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/73c9ba21dc82c6e67d7bacfc9c8d1d35a306b9c3b1a5edd8d1e6b743c6f4f78e/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/3a5febe131296426986213f0a63d21197a8c5c8cb1d4c2f331a3fa525d634fe0/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/0f9e293fd40d79adfd73f09247724c8c794d70b02fe5ac638ca65c0a513b0dad/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/5ecae3413619968416ee4eaefc9f6ae8f5f96d6ca589aeba74b874cfa88b9903/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/a0402eac7b9cb325f7f7d850df8fefc7628c2120ee0e7c2b32d0640d22bf5c2d/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/b5ec7445ff92c8c7ec27dd7d3d3e143fe3d4b68415a235ae7e4bb2c75dec7583/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/43d898b9d24c6b0ee454144bbc66c9d335383089452b88f21e7085ff123dddea/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/f62c391d861c04d3b950408d8896cb51682236891edc4ee1fc38c3853e5a5c81/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/7fa8d9133c04c2a1e4ce0513e1729929e603730d498daacc7e2918ae2e71137f/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/fee0db66893d578d5dc1051b166d4df40892b52f65cbf09fd5c43a5e181cb535/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/4ed2255010fe6f8c6f0cf461919d7dae167188a5e5e0c86822d91feab5a76d30/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/8f04096cee8a8f33946e8c2cc9b8029f2fe2b8ed11ce0166269bdc15e6e16d64/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/22023134d59d728f49f9b12db5fb2e0ab4a297dd7d0b969d4e52343286fe316e/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/e83b7d6ceba053448f24f42e6b4fa53b2b2a470dfe5130d27f7052d1e0384027/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/fa6a06f1d65b1558e32b27f037ae8e835bd0d81dc42b5e10dd77ae3648aa27fc/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/c6986a8f660bf891854bfb35f1706a352494877ba5210683eab6e5234f8dd991/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/5e578268a7ae829804e1196d50829de9a65fb590132c7ff6a75a89c597d5e138/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/12694c9597adbd1c3d7f1391389118e4125d08b2391a318db0604805c8e5b1fa/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/eb440adc90334a301651537cc382190b0ad8276717abfcbad905d4f4524860d0/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/efe6a89628889061d5ae1d843e2940675cc388b002ad1e2d52ba2e8009d41b4f/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/2c4ddba7bd8228dc66a7f274d15cf295934a00f5b5bea3156deeb053ac9e7f24/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/cabdf33f54874d681fd73e6e024e785b0619094f300af9bfb0fa545fb2e6003d/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/873857f09bcca4d7c64ba0416224e75a0f4ca1c1d666bfe2bbcaa17c683b167b/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/d70431e071a8e4d4ff322e93d7f7b65c08ddad502f1b8f17a7d1c8f86c93befa/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/66b23b609b79535848c3f3601afb3a2879b0fd5c102fca5185eea909bfd6cc2d/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/df0ad04b99f8488f937286e48583e951d46e2e407081b1e753ab34e83b349cfb/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/03d77a4d64a2ddbbe6c3719c8035ce458808827e2f6ba41962846e4c814676de/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9b93d4822a2e72b312f885eaf262c57ab47668ab1d6c18ad53006fbf4b8f39e5/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/da413fc86db046f8d7a1bea8515a9e1df78be3a56abfad5fe8dbb2a7a427b9ac/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/2c310d6f724dfba264db570595f8253a075f5d895d6d7c328ecfb5c745c6a931/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0ee8d80ea8da510ccbad65d6cff67002ef746a9d5c201bcbfc458a9dfd3f13ac/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/09e5d1535f5b65257c787945961b7e80bdf952d0310857402350ded00bf11728/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6b1671708e357c8e78c2af1f5b2dad6b4f0a753bf33419a26493ee39cf04d389/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7ff38bc22656662c5a50ad3bb1447d3473cf7e4468b2552c3eb4723cc6e391cd/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 07:28:55.922837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.922378    5829 manager.go:217] Machine: {Timestamp:2026-10-31 07:28:55.921845455 +0000 UTC m=+0.061006034 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100108 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ebf9231d6dbdcc605db03d1dffdd3 SystemUUID:ec2ebf92-31d6-dbdc-c605-db03d1dffdd3 BootID:dc38b392-66b2-4898-8795-69410b88ea50 Filesystems:[{Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dcc41609851bea2369d498de074702337e8bdda82cb0dff5822362ffc4d85974/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/658d33ef-d90b-448b-b4d0-e175b77a187f/volumes/kubernetes.io~projected/kube-api-access-mpnv5 DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/95b1f652-e473-4b0e-a9da-ac32940f95a8/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b72f45ccc31af5137314f6e552b5db1ff3eae84c2e41fcc5cd2b38593ebb6cee/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cce489912507a7128eabc59d8e95f8d1145bf3fb49f84ca8f2639aefba65e84b/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a1106ff94c3a8aa7b24c8e86d66006d4fa4e9d6ac46c1f682b5eee6af2ed14e/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea/volumes/kubernetes.io~projected/kube-api-access-w6l87 DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/836cf52b-b0e8-4e98-a812-21d0cebfaf63/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836cf52b-b0e8-4e98-a812-21d0cebfaf63/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7/volumes/kubernetes.io~projected/kube-api-access-nsnkc DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cee91069-0b51-442c-a3c3-71e71302b0e5/volumes/kubernetes.io~projected/kube-api-access-lkpsk DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:433 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:432 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/457dde04-3e55-4c30-a1c5-d13672f48a4a/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f4a456-26c2-4a45-bd30-b6795d5c03cc/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:474 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c86cae4f41950f6787ed94be7d254929c4cc1502a26a82f8aa50e4af83030c5d/userdata/shm DeviceMajor:0 DeviceMinor:484 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b62296f1-d84a-4dd1-8eb0-9f64567af021/volumes/kubernetes.io~projected/kube-api-access-52lps DeviceMajor:0 DeviceMinor:206 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8856b22a-b791-4ca1-8083-650b391429af/volumes/kubernetes.io~projected/kube-api-access-h4944 DeviceMajor:0 DeviceMinor:479 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/db612775-57b2-405c-a403-8c78f173d018/volumes/kubernetes.io~projected/kube-api-access-g7vps DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:320 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/658d33ef-d90b-448b-b4d0-e175b77a187f/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~projected/kube-api-access-x2wzb DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:427 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c24134f727811f6856dab38b787722771adb75015ff707ed8b73e7e7ecfe53f2/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:362 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26363f8b3ca96c3d4cfa68704cb208679ae1c4f559b96ca316317e75c9fdbe28/userdata/shm DeviceMajor:0 DeviceMinor:477 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/132e4deb-32cc-4b7b-a179-44a9700f1bac/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba447568733a33979f88bcdce24ef1ace1d241fe20e8b32a37135e2d1ba9a0c9/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:339 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:432 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70f074b267a802533fc01318aebb2b926b621bfa14a701947086928a0c04e779/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:430 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/95b1f652-e473-4b0e-a9da-ac32940f95a8/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cee91069-0b51-442c-a3c3-71e71302b0e5/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:470 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4fc187c9-2163-4de8-afbb-77f65de777c8/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da6208239c5f55b66611f1e47bb739d135e8abef821c0c42f3919c7c468f978e/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e3a7ead-4527-476b-9b02-c50e68e80166/volumes/kubernetes.io~projected/kube-api-access-cx78k DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:279 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e9d25f1c3282d0520135486e3cfbe329989f8d3152940e6988a10910b6e59f4c/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c50ba8c8f775968ea83b50a1cffe3e94b895e7dd2c0f6d7da6fc47edcb573d2/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:340 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:429 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/943a503a9bc0ee6a374d29505f320454a217981af0d72d26163429f181391af3/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/95b1f652-e473-4b0e-a9da-ac32940f95a8/volumes/kubernetes.io~projected/kube-api-access-kxz6l DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:318 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e5f9a7f1b210fbb830f1e68ff7c0dd62636fa7e0ee9ecb050eb680ccd8f99a5/userdata/shm DeviceMajor:0 DeviceMinor:438 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/836cf52b-b0e8-4e98-a812-21d0cebfaf63/volumes/kubernetes.io~projected/kube-api-access-66fgg DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:435 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4135f17c15b08ddff40838d0874e901eb5560c4d09c07d2e62eb661c88c198cb/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f206a32f-819d-4e61-9040-790e8c9183ec/volumes/kubernetes.io~projected/kube-api-access-xbx2j DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2af799e3-eb79-41f6-be34-411ea2f10326/volumes/kubernetes.io~projected/kube-api-access-cvhrd DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~projected/kube-api-access-hgglw DeviceMajor:0 DeviceMinor:323 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db53f4ce4263659743fb99007a507a0a5ee3401bf56259ba3de1b22515d21de0/userdata/shm DeviceMajor:0 DeviceMinor:378 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b922b9e6f40c1835b16f980b82d23e256f7f2527e59a0553cd0804c32f83ea8/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:321 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:422 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4fc187c9-2163-4de8-afbb-77f65de777c8/volumes/kubernetes.io~projected/kube-api-access-4r8cv DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e23e9f7f6762228112a7c326718a42286611483fda8838fa797b01cbf9c19090/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/457dde04-3e55-4c30-a1c5-d13672f48a4a/volumes/kubernetes.io~projected/kube-api-access-bcflp DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/658d33ef-d90b-448b-b4d0-e175b77a187f/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~projected/kube-api-access-cq5nl DeviceMajor:0 DeviceMinor:437 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/826e797743f1a8ab0415c3d272c61f0319adfa5509a314b51673345fe47f394d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:431 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c516d07a-8d53-4cae-9f4e-8b07114916b8/volumes/kubernetes.io~projected/kube-api-access-qfg4b DeviceMajor:0 DeviceMinor:341 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:432 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b62296f1-d84a-4dd1-8eb0-9f64567af021/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed8c10ee633e1ad18b4096e6c91ded20f77169bd3a89deab0b0c90b981179749/userdata/shm DeviceMajor:0 DeviceMinor:342 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:432 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8af93f76a20555c304f3ca74d05a669c2d2c3e652cb2b90cd7514f143b58965/userdata/shm DeviceMajor:0 DeviceMinor:475 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b10279343c85685543df2671b87124ea3283151b3c3ce9771b68f6fc1bbdba49/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/volumes/kubernetes.io~projected/kube-api-access-cffpp DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8401c67bdc047a50ee8712abf8fc395becfaa60b763a3985fab0d4d539031cc9/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:434 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4fc187c9-2163-4de8-afbb-77f65de777c8/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5/volumes/kubernetes.io~projected/kube-api-access-6tr9c DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f252c1a4-f927-4c73-9521-93f5cfbf2dc4/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:317 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/volumes/kubernetes.io~projected/kube-api-access-m7b5w DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3667fbda52b872511daaaf50ba7c878c6daadd2b38b2969636c77e8673b2e02/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c0fd0630-ea16-413c-a195-ff4cd58d8d21/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:284 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:428 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:26363f8b3ca96c3 MacAddress:5e:2c:96:63:e4:57 Speed:10000 Mtu:8901} {Name:8401c67bdc047a5 MacAddress:72:ee:ed:85:48:fa Speed:10000 Mtu:8901} {Name:8e5f9a7f1b210fb MacAddress:7e:15:e2:94:65:20 Speed:10000 Mtu:8901} {Name:943a503a9bc0ee6 MacAddress:8e:ce:a5:cb:99:62 Speed:10000 Mtu:8901} {Name:b8af93f76a20555 MacAddress:d2:84:b4:49:06:d7 Speed:10000 Mtu:8901} {Name:ba447568733a339 MacAddress:5a:f2:ff:03:c7:22 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ee:1f:1f:a3 Speed:0 Mtu:9001} {Name:br-int MacAddress:5a:0e:b6:bf:a6:1d Speed:0 Mtu:8901} {Name:c24134f727811f6 MacAddress:d2:f7:8e:ce:64:f5 Speed:10000 Mtu:8901} {Name:c86cae4f41950f6 MacAddress:3a:bf:70:32:70:91 Speed:10000 Mtu:8901} {Name:d3667fbda52b872 MacAddress:c2:59:9f:8e:99:c9 Speed:10000 Mtu:8901} {Name:da6208239c5f55b MacAddress:5e:f7:58:43:0a:e3 Speed:10000 Mtu:8901} {Name:db53f4ce4263659 MacAddress:9e:40:e7:b8:20:64 Speed:10000 Mtu:8901} {Name:e9d25f1c3282d05 MacAddress:8a:61:d0:47:48:d8 Speed:10000 Mtu:8901} {Name:ed8c10ee633e1ad MacAddress:8e:65:de:7b:12:60 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ee:1f:1f:a3 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:66:d6:e3:3d:41:bd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:a2:f0:97:a1:5e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 07:28:55.923159 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.922840    5829 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 07:28:55.923159 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.922889    5829 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 07:28:55.923350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.923328    5829 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 07:28:55.923529 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.923351    5829 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 07:28:55.923605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.923532    5829 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 07:28:55.923605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.923545    5829 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 07:28:55.923605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.923559    5829 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:28:55.923605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.923581    5829 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 07:28:55.923780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.923712    5829 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 07:28:55.923985 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.923971    5829 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 07:28:55.924125 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.924114    5829 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 07:28:55.924178 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.924164    5829 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 07:28:55.924226 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.924187    5829 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 07:28:55.924226 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.924207    5829 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 07:28:55.924226 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.924221    5829 kubelet.go:405] "Adding apiserver pod source"
Oct 31 07:28:55.924378 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.924313    5829 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 07:28:55.926409 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.926352    5829 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 07:28:55.926409 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.926370    5829 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:28:55.927087 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.927064    5829 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 07:28:55.929701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.929687    5829 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 07:28:55.929701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.929700    5829 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 07:28:55.929811 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.929801    5829 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 07:28:55.930010 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.929999    5829 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 07:28:55.930040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930028    5829 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 07:28:55.930127 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930118    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 07:28:55.930157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930132    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 07:28:55.930157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930138    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 07:28:55.930157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930144    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 07:28:55.930157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930150    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 07:28:55.930157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930155    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 07:28:55.930325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930161    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 07:28:55.930325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930167    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 07:28:55.930325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930174    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 07:28:55.930325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930179    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 07:28:55.930325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930188    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 07:28:55.930325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930198    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 07:28:55.930325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930226    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 07:28:55.930325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.930232    5829 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 07:28:55.933614 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.933599    5829 server.go:1263] "Started kubelet"
Oct 31 07:28:55.933704 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.933677    5829 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 07:28:55.933766 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.933707    5829 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 07:28:55.933817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.933795    5829 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 07:28:55.934167 ip-10-0-140-236 systemd[1]: Started Kubernetes Kubelet.
Oct 31 07:28:55.934629 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.934378    5829 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 07:28:55.935950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.935935    5829 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 07:28:55.942641 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.942616    5829 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 07:28:55.942861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.942815    5829 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 07:28:55.942952 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:28:55.942900    5829 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-236.ec2.internal\" not found"
Oct 31 07:28:55.942952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.942924    5829 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 07:28:55.942952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.942938    5829 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 07:28:55.943105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.942941    5829 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:22:46 +0000 UTC" deadline="2028-06-19 14:14:11.943548723 +0000 UTC"
Oct 31 07:28:55.943105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.942980    5829 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14334h45m16.000572112s"
Oct 31 07:28:55.943718 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.943376    5829 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 07:28:55.946879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.945800    5829 factory.go:55] Registering systemd factory
Oct 31 07:28:55.946879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.945823    5829 factory.go:223] Registration of the systemd container factory successfully
Oct 31 07:28:55.946879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.946051    5829 factory.go:153] Registering CRI-O factory
Oct 31 07:28:55.946879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.946067    5829 factory.go:223] Registration of the crio container factory successfully
Oct 31 07:28:55.946879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.946126    5829 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 07:28:55.946879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.946158    5829 factory.go:103] Registering Raw factory
Oct 31 07:28:55.946879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.946184    5829 manager.go:1199] Started watching for new ooms in manager
Oct 31 07:28:55.946879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.946571    5829 manager.go:319] Starting recovery of all containers
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947152    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-web-config" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947203    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c516d07a-8d53-4cae-9f4e-8b07114916b8" volumeName="kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947220    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2af799e3-eb79-41f6-be34-411ea2f10326" volumeName="kubernetes.io/projected/2af799e3-eb79-41f6-be34-411ea2f10326-kube-api-access-cvhrd" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947234    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836cf52b-b0e8-4e98-a812-21d0cebfaf63" volumeName="kubernetes.io/projected/836cf52b-b0e8-4e98-a812-21d0cebfaf63-kube-api-access-66fgg" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947247    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947259    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config-out" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947272    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947307    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-grpc-tls" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947324    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c516d07a-8d53-4cae-9f4e-8b07114916b8" volumeName="kubernetes.io/projected/c516d07a-8d53-4cae-9f4e-8b07114916b8-kube-api-access-qfg4b" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947336    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f252c1a4-f927-4c73-9521-93f5cfbf2dc4" volumeName="kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947349    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="658d33ef-d90b-448b-b4d0-e175b77a187f" volumeName="kubernetes.io/configmap/658d33ef-d90b-448b-b4d0-e175b77a187f-metrics-client-ca" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947362    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95b1f652-e473-4b0e-a9da-ac32940f95a8" volumeName="kubernetes.io/projected/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-api-access-kxz6l" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947375    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 31 07:28:55.947385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947391    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e3a7ead-4527-476b-9b02-c50e68e80166" volumeName="kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-daemon-config" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947408    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e3a7ead-4527-476b-9b02-c50e68e80166" volumeName="kubernetes.io/projected/1e3a7ead-4527-476b-9b02-c50e68e80166-kube-api-access-cx78k" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947421    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" volumeName="kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947435    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc187c9-2163-4de8-afbb-77f65de777c8" volumeName="kubernetes.io/secret/4fc187c9-2163-4de8-afbb-77f65de777c8-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947449    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95b1f652-e473-4b0e-a9da-ac32940f95a8" volumeName="kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947461    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7" volumeName="kubernetes.io/projected/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-kube-api-access-nsnkc" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947474    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b62296f1-d84a-4dd1-8eb0-9f64567af021" volumeName="kubernetes.io/projected/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-api-access-52lps" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947488    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-cluster-tls-config" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947500    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-web-config" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947511    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="457dde04-3e55-4c30-a1c5-d13672f48a4a" volumeName="kubernetes.io/secret/457dde04-3e55-4c30-a1c5-d13672f48a4a-metrics-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947521    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95b1f652-e473-4b0e-a9da-ac32940f95a8" volumeName="kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-metrics-client-ca" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947533    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947548    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947560    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-kube-api-access-cq5nl" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947571    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-metrics-client-certs" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947588    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c516d07a-8d53-4cae-9f4e-8b07114916b8" volumeName="kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947601    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc187c9-2163-4de8-afbb-77f65de777c8" volumeName="kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-config" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947613    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b62296f1-d84a-4dd1-8eb0-9f64567af021" volumeName="kubernetes.io/empty-dir/b62296f1-d84a-4dd1-8eb0-9f64567af021-data-volume" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947625    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b62296f1-d84a-4dd1-8eb0-9f64567af021" volumeName="kubernetes.io/secret/b62296f1-d84a-4dd1-8eb0-9f64567af021-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947653    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-main-db" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947679    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c516d07a-8d53-4cae-9f4e-8b07114916b8" volumeName="kubernetes.io/empty-dir/c516d07a-8d53-4cae-9f4e-8b07114916b8-audit-log" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947690    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee91069-0b51-442c-a3c3-71e71302b0e5" volumeName="kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947702    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea" volumeName="kubernetes.io/projected/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-kube-api-access-w6l87" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947748    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f252c1a4-f927-4c73-9521-93f5cfbf2dc4" volumeName="kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947764    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2af799e3-eb79-41f6-be34-411ea2f10326" volumeName="kubernetes.io/configmap/2af799e3-eb79-41f6-be34-411ea2f10326-iptables-alerter-script" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947777    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee" volumeName="kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-metrics-client-ca" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947795    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-metrics-client-ca" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947807    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c516d07a-8d53-4cae-9f4e-8b07114916b8" volumeName="kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-client-ca-bundle" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947820    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db612775-57b2-405c-a403-8c78f173d018" volumeName="kubernetes.io/projected/db612775-57b2-405c-a403-8c78f173d018-kube-api-access-g7vps" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947834    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f252c1a4-f927-4c73-9521-93f5cfbf2dc4" volumeName="kubernetes.io/projected/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-kube-api-access-hgglw" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947847    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f252c1a4-f927-4c73-9521-93f5cfbf2dc4" volumeName="kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947859    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f252c1a4-f927-4c73-9521-93f5cfbf2dc4" volumeName="kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947871    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5" volumeName="kubernetes.io/secret/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-cert" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947883    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee" volumeName="kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947898    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95b1f652-e473-4b0e-a9da-ac32940f95a8" volumeName="kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947909    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947921    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f4a456-26c2-4a45-bd30-b6795d5c03cc" volumeName="kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947933    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f252c1a4-f927-4c73-9521-93f5cfbf2dc4" volumeName="kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-grpc-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947945    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee" volumeName="kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947957    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="457dde04-3e55-4c30-a1c5-d13672f48a4a" volumeName="kubernetes.io/configmap/457dde04-3e55-4c30-a1c5-d13672f48a4a-config-volume" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947969    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7" volumeName="kubernetes.io/empty-dir/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-tmp-dir" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.947984    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b62296f1-d84a-4dd1-8eb0-9f64567af021" volumeName="kubernetes.io/configmap/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948000    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948027    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f252c1a4-f927-4c73-9521-93f5cfbf2dc4" volumeName="kubernetes.io/empty-dir/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-tmp" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948039    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="132e4deb-32cc-4b7b-a179-44a9700f1bac" volumeName="kubernetes.io/secret/132e4deb-32cc-4b7b-a179-44a9700f1bac-agent-certs" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948053    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="658d33ef-d90b-448b-b4d0-e175b77a187f" volumeName="kubernetes.io/projected/658d33ef-d90b-448b-b4d0-e175b77a187f-kube-api-access-mpnv5" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948066    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836cf52b-b0e8-4e98-a812-21d0cebfaf63" volumeName="kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-tmp" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948077    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95b1f652-e473-4b0e-a9da-ac32940f95a8" volumeName="kubernetes.io/empty-dir/95b1f652-e473-4b0e-a9da-ac32940f95a8-volume-directive-shadow" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948091    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948105    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8856b22a-b791-4ca1-8083-650b391429af" volumeName="kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948117    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="95b1f652-e473-4b0e-a9da-ac32940f95a8" volumeName="kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948131    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-out" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948143    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948155    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948168    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948181    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948193    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c516d07a-8d53-4cae-9f4e-8b07114916b8" volumeName="kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948205    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="132e4deb-32cc-4b7b-a179-44a9700f1bac" volumeName="kubernetes.io/configmap/132e4deb-32cc-4b7b-a179-44a9700f1bac-konnectivity-ca" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948219    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" volumeName="kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948231    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc187c9-2163-4de8-afbb-77f65de777c8" volumeName="kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948244    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc187c9-2163-4de8-afbb-77f65de777c8" volumeName="kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-env-overrides" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948255    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-kube-api-access-x2wzb" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948266    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948295    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948309    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c516d07a-8d53-4cae-9f4e-8b07114916b8" volumeName="kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948323    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5" volumeName="kubernetes.io/projected/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-kube-api-access-6tr9c" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948339    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e3a7ead-4527-476b-9b02-c50e68e80166" volumeName="kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-cni-binary-copy" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948353    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="457dde04-3e55-4c30-a1c5-d13672f48a4a" volumeName="kubernetes.io/empty-dir/457dde04-3e55-4c30-a1c5-d13672f48a4a-tmp-dir" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948363    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="457dde04-3e55-4c30-a1c5-d13672f48a4a" volumeName="kubernetes.io/projected/457dde04-3e55-4c30-a1c5-d13672f48a4a-kube-api-access-bcflp" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948378    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" volumeName="kubernetes.io/projected/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-kube-api-access-m7b5w" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948391    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="658d33ef-d90b-448b-b4d0-e175b77a187f" volumeName="kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948403    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="658d33ef-d90b-448b-b4d0-e175b77a187f" volumeName="kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948419    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-tls-assets" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948434    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948446    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" volumeName="kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-binary-copy" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948459    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fc187c9-2163-4de8-afbb-77f65de777c8" volumeName="kubernetes.io/projected/4fc187c9-2163-4de8-afbb-77f65de777c8-kube-api-access-4r8cv" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948470    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-db" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948498    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-tls-assets" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948510    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cee91069-0b51-442c-a3c3-71e71302b0e5" volumeName="kubernetes.io/projected/cee91069-0b51-442c-a3c3-71e71302b0e5-kube-api-access-lkpsk" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948522    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f206a32f-819d-4e61-9040-790e8c9183ec" volumeName="kubernetes.io/projected/f206a32f-819d-4e61-9040-790e8c9183ec-kube-api-access-xbx2j" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948532    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f252c1a4-f927-4c73-9521-93f5cfbf2dc4" volumeName="kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948544    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948553    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea" volumeName="kubernetes.io/secret/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-monitoring-plugin-cert" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948565    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f252c1a4-f927-4c73-9521-93f5cfbf2dc4" volumeName="kubernetes.io/configmap/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-metrics-client-ca" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948578    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee" volumeName="kubernetes.io/empty-dir/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-textfile" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948589    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee" volumeName="kubernetes.io/projected/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-kube-api-access-cffpp" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948602    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="836cf52b-b0e8-4e98-a812-21d0cebfaf63" volumeName="kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-tuned" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948614    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-volume" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948626    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f206a32f-819d-4e61-9040-790e8c9183ec" volumeName="kubernetes.io/configmap/f206a32f-819d-4e61-9040-790e8c9183ec-serviceca" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948637    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee" volumeName="kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-tls" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948650    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0fd0630-ea16-413c-a195-ff4cd58d8d21" volumeName="kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948662    5829 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" volumeName="kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948673    5829 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 07:28:55.949662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.948679    5829 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 07:28:55.954079 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.953023    5829 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 07:28:55.966144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.966116    5829 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 07:28:55.966144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.966145    5829 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 07:28:55.966433 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.966167    5829 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 07:28:55.966433 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:28:55.966207    5829 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 07:28:55.977080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.977046    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:28:55.979893 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.979846    5829 generic.go:356] "Generic (PLEG): container finished" podID="3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee" containerID="19224fd67bb411f5258a5a864f86fc687c3750b9a104f5eb426cb90a76f2c257" exitCode=0
Oct 31 07:28:55.980521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.980498    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:28:55.986975 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.986952    5829 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 07:28:55.987113 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.987093    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:28:55.987531 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.987504    5829 generic.go:356] "Generic (PLEG): container finished" podID="4fc187c9-2163-4de8-afbb-77f65de777c8" containerID="7adce90669d2aad149ed39c047a8cc97f157b1cdd1dc4e112adb771e5e397eec" exitCode=1
Oct 31 07:28:55.987609 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.987593    5829 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:55.994329 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.994302    5829 generic.go:356] "Generic (PLEG): container finished" podID="c4bf6b3f-71ad-4321-9bbf-4d713e4f2110" containerID="c16f93a5d8e8d7102bbb53c3e3258b9844b4c26a046a90dfafaff93278cdd944" exitCode=0
Oct 31 07:28:55.999335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.999315    5829 generic.go:356] "Generic (PLEG): container finished" podID="c0fd0630-ea16-413c-a195-ff4cd58d8d21" containerID="2d31e1e743d4a40168f2f592ffbcf2761d4957f3e2cd7b79fb3172f3d43a26da" exitCode=0
Oct 31 07:28:55.999450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:55.999435    5829 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:56.000858 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.000838    5829 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:56.012218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.012198    5829 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 07:28:56.014842 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.014818    5829 generic.go:356] "Generic (PLEG): container finished" podID="60bef5910d750df47fec5174fe909f02" containerID="f7e2cee5c0878f4e2d83dec395209141fd051820c005561f91b8a9e8dcf0f019" exitCode=0
Oct 31 07:28:56.029412 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.029381    5829 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="2c4b8aea403e45be5e7fe64e0e799285abb0e7b5fbd7aa939a177bddd02566b1" exitCode=0
Oct 31 07:28:56.029412 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.029410    5829 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="4265ed8ba65349630249001f218f5905b09a361ebebaba3c4f149bd117576afc" exitCode=0
Oct 31 07:28:56.029412 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.029417    5829 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="d4b0181eafb80a7a545229514c8226f04d227637373b6fa92d84567b01eed01e" exitCode=0
Oct 31 07:28:56.029649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.029423    5829 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="e6d10b6029d7e61aca133c7bf586726749005f14c5c0f73499dbf9c65c23b844" exitCode=0
Oct 31 07:28:56.029649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.029428    5829 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="441ab54ecf9418722396958c40f2d03efebede35ffcede7a2e6a5728f9a102f2" exitCode=0
Oct 31 07:28:56.029649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.029433    5829 generic.go:356] "Generic (PLEG): container finished" podID="486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb" containerID="26a555ea3cfb53e094af4d2c710104a1cac581a108671596c4f49ff4affb1c68" exitCode=0
Oct 31 07:28:56.065817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.065741    5829 manager.go:324] Recovery completed
Oct 31 07:28:56.066320 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:28:56.066300    5829 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 07:28:56.090121 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090093    5829 cpu_manager.go:226] "Starting" policy="none"
Oct 31 07:28:56.090121 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090112    5829 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 07:28:56.090298 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090134    5829 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 07:28:56.090332 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090305    5829 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 07:28:56.090455 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090429    5829 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 07:28:56.090485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090455    5829 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 07:28:56.090485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090461    5829 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 07:28:56.090485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090467    5829 policy_none.go:50] "Start"
Oct 31 07:28:56.090485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090478    5829 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 07:28:56.090605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090491    5829 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 07:28:56.090645 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090635    5829 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 07:28:56.090674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090645    5829 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 07:28:56.090674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.090651    5829 policy_none.go:44] "Start"
Oct 31 07:28:56.095452 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.095422    5829 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 07:28:56.099108 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.099093    5829 manager.go:341] "Starting Device Plugin manager"
Oct 31 07:28:56.099646 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:28:56.099118    5829 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 07:28:56.099646 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.099647    5829 server.go:87] "Starting device plugin registration server"
Oct 31 07:28:56.099953 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.099943    5829 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 07:28:56.099984 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.099954    5829 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 07:28:56.100069 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.100035    5829 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 07:28:56.100446 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.100145    5829 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 07:28:56.100527 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.100471    5829 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 07:28:56.100527 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.100480    5829 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 07:28:56.100654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.100552    5829 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:28:56.10045589Z","UUID":"e51a9d30-00ab-4fca-814f-64bebaa6b4eb","Handler":null,"Name":"","Endpoint":""}
Oct 31 07:28:56.102604 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:28:56.102579    5829 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 07:28:56.102763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.102744    5829 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 07:28:56.102878 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.102805    5829 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 07:28:56.203705 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.203671    5829 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 07:28:56.206023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.206003    5829 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientMemory"
Oct 31 07:28:56.206128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.206035    5829 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasNoDiskPressure"
Oct 31 07:28:56.206128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.206045    5829 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-236.ec2.internal" event="NodeHasSufficientPID"
Oct 31 07:28:56.206128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.206094    5829 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.240401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.240377    5829 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.240532 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.240455    5829 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.266884 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.266826    5829 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"]
Oct 31 07:28:56.267525 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.267486    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" event={"ID":"e82d0dbde7394ec25631244e1f1d5cd9","Type":"ContainerStarted","Data":"61602b9beadcf2a2d4f684dd3032c13a671318b6ea65036cc5c326dbb4d7c562"}
Oct 31 07:28:56.267616 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.267535    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal" event={"ID":"e82d0dbde7394ec25631244e1f1d5cd9","Type":"ContainerStarted","Data":"8a1106ff94c3a8aa7b24c8e86d66006d4fa4e9d6ac46c1f682b5eee6af2ed14e"}
Oct 31 07:28:56.267616 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.267558    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerStarted","Data":"5afc719f006c15a4a9bdfc29b47b7d25b81e1cb14bc0e6113120b60fa0d4b8cd"}
Oct 31 07:28:56.267616 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.267565    5829 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.267728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.267567    5829 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.267728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.267571    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerDied","Data":"f7e2cee5c0878f4e2d83dec395209141fd051820c005561f91b8a9e8dcf0f019"}
Oct 31 07:28:56.267782 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.267752    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal" event={"ID":"60bef5910d750df47fec5174fe909f02","Type":"ContainerStarted","Data":"cce489912507a7128eabc59d8e95f8d1145bf3fb49f84ca8f2639aefba65e84b"}
Oct 31 07:28:56.291806 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.291780    5829 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:56.291932 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:28:56.291836    5829 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.292069 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.292053    5829 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:56.292104 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:28:56.292092    5829 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.389439 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.389413    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.389569 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.389459    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.389569 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.389486    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.490027 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.489993    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.490161 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.490036    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.490161 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.490056    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.490161 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.490105    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.490161 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.490120    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60bef5910d750df47fec5174fe909f02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" (UID: \"60bef5910d750df47fec5174fe909f02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.490338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.490170    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e82d0dbde7394ec25631244e1f1d5cd9-config\") pod \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" (UID: \"e82d0dbde7394ec25631244e1f1d5cd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:56.925019 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.924983    5829 apiserver.go:52] "Watching apiserver"
Oct 31 07:28:56.980646 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.980610    5829 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 07:28:56.981267 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.981243    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal","openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t","openshift-network-diagnostics/network-check-target-zb9td","openshift-multus/multus-5jj86","openshift-network-operator/iptables-alerter-96gd4","openshift-dns/dns-default-hzqlg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl","openshift-ingress-canary/ingress-canary-9lgfj","openshift-monitoring/metrics-server-b67d5b6c9-897nz","openshift-dns/node-resolver-v59hm","openshift-image-registry/node-ca-wbg8z","openshift-ovn-kubernetes/ovnkube-node-4g4jz","kube-system/global-pull-secret-syncer-82457","openshift-cluster-node-tuning-operator/tuned-86nzk","openshift-monitoring/alertmanager-main-0","openshift-monitoring/node-exporter-hbnc6","openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf","openshift-multus/multus-additional-cni-plugins-cc6gh","openshift-multus/network-metrics-daemon-s2r9l","kube-system/konnectivity-agent-pkj5d","kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal","openshift-insights/insights-runtime-extractor-525tp","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf","openshift-monitoring/thanos-querier-8cd777578-wbgnk"]
Oct 31 07:28:56.998609 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998582    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 07:28:56.998755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998689    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 07:28:56.998755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998705    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 07:28:56.998755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998743    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:56.998876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998807    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 07:28:56.998876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998829    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 07:28:56.998876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998855    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 07:28:56.999028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998930    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 07:28:56.999028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998962    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 07:28:56.999028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998973    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 07:28:56.999028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998970    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:56.999028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998988    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 07:28:56.999028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.998964    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 07:28:56.999204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999030    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nzlr\""
Oct 31 07:28:56.999204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999039    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 07:28:56.999204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999051    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:28:56.999204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999031    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-548vl\""
Oct 31 07:28:56.999426 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999412    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gjb5j\""
Oct 31 07:28:56.999489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999461    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x6ssn\""
Oct 31 07:28:56.999563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999549    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 07:28:56.999597 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999554    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7m4f\""
Oct 31 07:28:56.999633 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999603    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 07:28:56.999668 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999659    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgg5g\""
Oct 31 07:28:56.999730 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999717    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7l5b\""
Oct 31 07:28:56.999797 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999782    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 07:28:56.999866 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:56.999854    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xhs6v\""
Oct 31 07:28:57.008636 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.008616    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 07:28:57.009042 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.009013    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 07:28:57.009138 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.009108    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wkhqf\""
Oct 31 07:28:57.009895 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.009883    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgtls\""
Oct 31 07:28:57.009971 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.009958    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 07:28:57.010015 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.009984    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dql9j\""
Oct 31 07:28:57.010253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.010230    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qk9pc\""
Oct 31 07:28:57.010712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.010697    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 07:28:57.010769 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.010726    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 07:28:57.010817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.010787    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 07:28:57.010987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.010973    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 07:28:57.011110 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011096    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 07:28:57.011299 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011221    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 07:28:57.011299 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011222    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 07:28:57.011404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011238    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 07:28:57.011404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011240    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 07:28:57.011404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011242    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 07:28:57.011404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011255    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 07:28:57.011404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011260    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh9bq\""
Oct 31 07:28:57.011404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011268    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 07:28:57.011404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011390    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011422    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011443    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011477    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011480    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011501    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011518    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011532    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011535    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011549    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011566    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011647    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 07:28:57.011676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011665    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 07:28:57.012037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011706    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 07:28:57.012037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011710    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 07:28:57.012037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011750    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 07:28:57.012037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.011824    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 07:28:57.012240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012225    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 07:28:57.012473 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012447    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vhq6\""
Oct 31 07:28:57.012473 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012461    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 07:28:57.012578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012509    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ppvgk\""
Oct 31 07:28:57.012578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012521    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 07:28:57.012578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012512    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m2lq9\""
Oct 31 07:28:57.012578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012524    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6pt6b\""
Oct 31 07:28:57.012578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012554    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 07:28:57.012764 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012751    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xtmc6\""
Oct 31 07:28:57.012839 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012772    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 07:28:57.012985 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.012972    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cg7nk\""
Oct 31 07:28:57.013261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013246    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 07:28:57.013261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013254    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 07:28:57.013379 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013305    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 07:28:57.013379 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013314    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 07:28:57.013462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013445    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 07:28:57.013507 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013495    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 07:28:57.013554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013512    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 07:28:57.013554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013537    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 07:28:57.013632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013571    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 07:28:57.013632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013575    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 07:28:57.013632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013588    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 07:28:57.013632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013623    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 07:28:57.013765 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013638    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 07:28:57.013765 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013685    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 07:28:57.013841 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013779    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 07:28:57.013841 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013836    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 07:28:57.013939 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013844    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 07:28:57.013939 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013848    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 07:28:57.013939 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013885    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 07:28:57.013939 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013898    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-78uabpe1lkr2i\""
Oct 31 07:28:57.013939 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.013915    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fdnb6\""
Oct 31 07:28:57.014129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014025    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 07:28:57.014129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014057    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-oe1fmgg4dv9t\""
Oct 31 07:28:57.014129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014067    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8qb8s\""
Oct 31 07:28:57.014129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014080    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-58mwv\""
Oct 31 07:28:57.014129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014084    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 07:28:57.014129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014085    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 07:28:57.014129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014112    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 07:28:57.014401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014388    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 07:28:57.014446 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014402    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 07:28:57.014446 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.014393    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 07:28:57.022327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.022310    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1lcltrbisfsua\""
Oct 31 07:28:57.022408 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.022374    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x4pvw\""
Oct 31 07:28:57.026627 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.026606    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 07:28:57.027869 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.027850    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 07:28:57.027945 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.027911    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 07:28:57.036149 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.036129    5829 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:57.036654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.036638    5829 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:57.046268 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.046248    5829 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 07:28:57.057789 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.057774    5829 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:57.057858 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:28:57.057849    5829 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:57.057952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.057937    5829 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 07:28:57.057996 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:28:57.057987    5829 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-236.ec2.internal"
Oct 31 07:28:57.094246 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094208    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-var-lib-kubelet\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.094413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094253    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-tmp\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.094413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094274    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95b1f652-e473-4b0e-a9da-ac32940f95a8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.094413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094385    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/95b1f652-e473-4b0e-a9da-ac32940f95a8-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.094552 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094426    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-tmp\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.094552 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094427    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.094552 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094499    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-netd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.094552 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094520    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-script-lib\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.094552 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094536    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.094785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094561    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.094785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094610    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.094785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094643    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.094785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094646    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.094785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094680    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.094785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094681    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.094785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094727    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-tmp\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.094785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094758    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-tls\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.094785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094785    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkpsk\" (UniqueName: \"kubernetes.io/projected/cee91069-0b51-442c-a3c3-71e71302b0e5-kube-api-access-lkpsk\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094814    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-cnibin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094843    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-node-log\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094868    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-run\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094880    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094893    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-tmp-dir\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094892    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094730    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-script-lib\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.094971    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-tmp\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095000    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-tmp-dir\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095008    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/658d33ef-d90b-448b-b4d0-e175b77a187f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095083    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-tls\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.095170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095152    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095200    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-lib-modules\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095205    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095270    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/658d33ef-d90b-448b-b4d0-e175b77a187f-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095392    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-sys-fs\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095418    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095435    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7vps\" (UniqueName: \"kubernetes.io/projected/db612775-57b2-405c-a403-8c78f173d018-kube-api-access-g7vps\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095490    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095536    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095610    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095644    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095717    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-cert\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:57.095814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095789    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-out\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095826    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095883    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c516d07a-8d53-4cae-9f4e-8b07114916b8-audit-log\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095885    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095898    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-out\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095922    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-cert\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095939    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/132e4deb-32cc-4b7b-a179-44a9700f1bac-konnectivity-ca\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095966    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095982    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c516d07a-8d53-4cae-9f4e-8b07114916b8-audit-log\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.095982    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-config\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096083    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096129    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/132e4deb-32cc-4b7b-a179-44a9700f1bac-konnectivity-ca\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096131    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096199    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-ovnkube-config\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.096275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096249    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-metrics-server-audit-profiles\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096305    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096323    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096330    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-host\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096355    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096395    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-textfile\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096401    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096425    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-etc-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096451    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-multus-certs\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096475    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-textfile\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096493    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-metrics-server-audit-profiles\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096500    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysconfig\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096536    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-sys\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096556    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsnkc\" (UniqueName: \"kubernetes.io/projected/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-kube-api-access-nsnkc\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096586    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-root\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096602    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096619    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096633    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-cni-binary-copy\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096670    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-kubelet\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096692    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096709    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096709    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b62296f1-d84a-4dd1-8eb0-9f64567af021-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096747    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-systemd\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096776    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096803    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-cni-binary-copy\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.096838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096843    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-grpc-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096875    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096893    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096905    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-netns\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096920    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-binary-copy\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096970    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.096981    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b62296f1-d84a-4dd1-8eb0-9f64567af021-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097002    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-dbus\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097029    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-metrics-client-ca\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097067    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-kubelet\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097081    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-grpc-tls\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097102    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-bin\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097130    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097155    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-system-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097166    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-metrics-client-ca\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097171    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx78k\" (UniqueName: \"kubernetes.io/projected/1e3a7ead-4527-476b-9b02-c50e68e80166-kube-api-access-cx78k\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097187    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097222    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097239    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097256    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66fgg\" (UniqueName: \"kubernetes.io/projected/836cf52b-b0e8-4e98-a812-21d0cebfaf63-kube-api-access-66fgg\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097298    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/457dde04-3e55-4c30-a1c5-d13672f48a4a-metrics-tls\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097318    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097325    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097350    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5nl\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-kube-api-access-cq5nl\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097373    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgglw\" (UniqueName: \"kubernetes.io/projected/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-kube-api-access-hgglw\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097396    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097421    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-k8s-cni-cncf-io\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097451    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-ovn\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097477    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-kubelet-config\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097488    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097502    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097449    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097502    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097534    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/457dde04-3e55-4c30-a1c5-d13672f48a4a-metrics-tls\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097477    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097547    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tr9c\" (UniqueName: \"kubernetes.io/projected/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-kube-api-access-6tr9c\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097632    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/457dde04-3e55-4c30-a1c5-d13672f48a4a-config-volume\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097682    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config-out\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097709    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457dde04-3e55-4c30-a1c5-d13672f48a4a-tmp-dir\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097747    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cee91069-0b51-442c-a3c3-71e71302b0e5-metrics-certs\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097802    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097834    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457dde04-3e55-4c30-a1c5-d13672f48a4a-tmp-dir\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097841    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/457dde04-3e55-4c30-a1c5-d13672f48a4a-config-volume\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:57.097861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097881    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097901    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.097935    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098016    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-multus\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098068    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098098    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/132e4deb-32cc-4b7b-a179-44a9700f1bac-agent-certs\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098112    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098127    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-config-volume\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098148    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c516d07a-8d53-4cae-9f4e-8b07114916b8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098149    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2af799e3-eb79-41f6-be34-411ea2f10326-host-slash\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098194    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-wtmp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098217    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098256    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098315    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/132e4deb-32cc-4b7b-a179-44a9700f1bac-agent-certs\") pod \"konnectivity-agent-pkj5d\" (UID: \"132e4deb-32cc-4b7b-a179-44a9700f1bac\") " pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098345    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cnibin\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098382    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b62296f1-d84a-4dd1-8eb0-9f64567af021-crio-socket\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098445    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52lps\" (UniqueName: \"kubernetes.io/projected/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-api-access-52lps\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098479    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcflp\" (UniqueName: \"kubernetes.io/projected/457dde04-3e55-4c30-a1c5-d13672f48a4a-kube-api-access-bcflp\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098570    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-client-certs\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098598    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b62296f1-d84a-4dd1-8eb0-9f64567af021-data-volume\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098617    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhrd\" (UniqueName: \"kubernetes.io/projected/2af799e3-eb79-41f6-be34-411ea2f10326-kube-api-access-cvhrd\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098671    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbx2j\" (UniqueName: \"kubernetes.io/projected/f206a32f-819d-4e61-9040-790e8c9183ec-kube-api-access-xbx2j\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098683    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b62296f1-d84a-4dd1-8eb0-9f64567af021-data-volume\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098701    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-socket-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098760    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098802    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098831    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfg4b\" (UniqueName: \"kubernetes.io/projected/c516d07a-8d53-4cae-9f4e-8b07114916b8-kube-api-access-qfg4b\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098860    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-slash\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098865    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-secret-metrics-server-client-certs\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098884    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-log-socket\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098911    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-env-overrides\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098938    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpnv5\" (UniqueName: \"kubernetes.io/projected/658d33ef-d90b-448b-b4d0-e175b77a187f-kube-api-access-mpnv5\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098946    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.098981    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099010    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099037    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099064    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099091    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-etc-kubernetes\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099118    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-systemd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099141    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f206a32f-819d-4e61-9040-790e8c9183ec-host\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099165    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099095    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fc187c9-2163-4de8-afbb-77f65de777c8-env-overrides\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099234    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099330    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099360    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-client-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099372    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099385    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099432    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099437    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7b5w\" (UniqueName: \"kubernetes.io/projected/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-kube-api-access-m7b5w\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099481    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-var-lib-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099556    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-web-config\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099578    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8cv\" (UniqueName: \"kubernetes.io/projected/4fc187c9-2163-4de8-afbb-77f65de777c8-kube-api-access-4r8cv\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099609    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c516d07a-8d53-4cae-9f4e-8b07114916b8-client-ca-bundle\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099724    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f206a32f-819d-4e61-9040-790e8c9183ec-serviceca\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:28:57.099712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099745    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-tls\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099772    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099794    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099819    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-socket-dir-parent\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099834    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-systemd-units\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099871    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-kubernetes\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099915    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-conf\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.099968    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100007    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100035    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-tls\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100037    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100064    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100081    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-bin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100139    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100168    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100173    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxz6l\" (UniqueName: \"kubernetes.io/projected/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-api-access-kxz6l\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100198    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-daemon-config\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100199    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100232    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2wzb\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-kube-api-access-x2wzb\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100257    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100296    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-tuned\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100323    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-hosts-file\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100359    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100375    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/658d33ef-d90b-448b-b4d0-e175b77a187f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100386    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6l87\" (UniqueName: \"kubernetes.io/projected/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-kube-api-access-w6l87\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100396    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f206a32f-819d-4e61-9040-790e8c9183ec-serviceca\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100417    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-tuned\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100423    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-daemon-config\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100421    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-sys\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100544    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100612    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100668    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-netns\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100694    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100750    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100781    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100787    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100857    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2af799e3-eb79-41f6-be34-411ea2f10326-iptables-alerter-script\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100908    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100933    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-device-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100951    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.100970    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101037    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2af799e3-eb79-41f6-be34-411ea2f10326-iptables-alerter-script\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101096    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-modprobe-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101122    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101130    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101186    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc187c9-2163-4de8-afbb-77f65de777c8-ovn-node-metrics-cert\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101217    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-kube-api-access-cffpp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101244    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101274    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-hostroot\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101317    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101372    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101412    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-registration-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101514    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fc187c9-2163-4de8-afbb-77f65de777c8-ovn-node-metrics-cert\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101515    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-metrics-client-ca\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101555    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-web-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101583    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101609    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101639    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-conf-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101667    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101678    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-metrics-client-ca\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101696    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101786    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101869    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0fd0630-ea16-413c-a195-ff4cd58d8d21-web-config\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101873    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.101840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.101916    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.103761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.102050    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.103761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.102088    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fd0630-ea16-413c-a195-ff4cd58d8d21-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.103761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.102106    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-original-pull-secret\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:57.103761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.102875    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.113064 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.113038    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7vps\" (UniqueName: \"kubernetes.io/projected/db612775-57b2-405c-a403-8c78f173d018-kube-api-access-g7vps\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.113175 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.113079    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkpsk\" (UniqueName: \"kubernetes.io/projected/cee91069-0b51-442c-a3c3-71e71302b0e5-kube-api-access-lkpsk\") pod \"network-metrics-daemon-s2r9l\" (UID: \"cee91069-0b51-442c-a3c3-71e71302b0e5\") " pod="openshift-multus/network-metrics-daemon-s2r9l"
Oct 31 07:28:57.114098 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.114076    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsnkc\" (UniqueName: \"kubernetes.io/projected/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-kube-api-access-nsnkc\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:28:57.114796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.114777    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66fgg\" (UniqueName: \"kubernetes.io/projected/836cf52b-b0e8-4e98-a812-21d0cebfaf63-kube-api-access-66fgg\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.115993 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.115972    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4944\" (UniqueName: \"kubernetes.io/projected/8856b22a-b791-4ca1-8083-650b391429af-kube-api-access-h4944\") pod \"network-check-target-zb9td\" (UID: \"8856b22a-b791-4ca1-8083-650b391429af\") " pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:28:57.116134 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.116107    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgglw\" (UniqueName: \"kubernetes.io/projected/f252c1a4-f927-4c73-9521-93f5cfbf2dc4-kube-api-access-hgglw\") pod \"thanos-querier-8cd777578-wbgnk\" (UID: \"f252c1a4-f927-4c73-9521-93f5cfbf2dc4\") " pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:57.116250 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.116113    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5nl\" (UniqueName: \"kubernetes.io/projected/c4bf6b3f-71ad-4321-9bbf-4d713e4f2110-kube-api-access-cq5nl\") pod \"prometheus-k8s-0\" (UID: \"c4bf6b3f-71ad-4321-9bbf-4d713e4f2110\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:28:57.116319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.116253    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx78k\" (UniqueName: \"kubernetes.io/projected/1e3a7ead-4527-476b-9b02-c50e68e80166-kube-api-access-cx78k\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.116367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.116340    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tr9c\" (UniqueName: \"kubernetes.io/projected/0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5-kube-api-access-6tr9c\") pod \"ingress-canary-9lgfj\" (UID: \"0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5\") " pod="openshift-ingress-canary/ingress-canary-9lgfj"
Oct 31 07:28:57.116686 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.116669    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbx2j\" (UniqueName: \"kubernetes.io/projected/f206a32f-819d-4e61-9040-790e8c9183ec-kube-api-access-xbx2j\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:28:57.116985 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.116964    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfg4b\" (UniqueName: \"kubernetes.io/projected/c516d07a-8d53-4cae-9f4e-8b07114916b8-kube-api-access-qfg4b\") pod \"metrics-server-b67d5b6c9-897nz\" (UID: \"c516d07a-8d53-4cae-9f4e-8b07114916b8\") " pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:28:57.117375 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.117355    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52lps\" (UniqueName: \"kubernetes.io/projected/b62296f1-d84a-4dd1-8eb0-9f64567af021-kube-api-access-52lps\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.117450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.117375    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcflp\" (UniqueName: \"kubernetes.io/projected/457dde04-3e55-4c30-a1c5-d13672f48a4a-kube-api-access-bcflp\") pod \"dns-default-hzqlg\" (UID: \"457dde04-3e55-4c30-a1c5-d13672f48a4a\") " pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:28:57.117517 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.117497    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhrd\" (UniqueName: \"kubernetes.io/projected/2af799e3-eb79-41f6-be34-411ea2f10326-kube-api-access-cvhrd\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:28:57.117686 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.117672    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8cv\" (UniqueName: \"kubernetes.io/projected/4fc187c9-2163-4de8-afbb-77f65de777c8-kube-api-access-4r8cv\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.117810 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.117790    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpnv5\" (UniqueName: \"kubernetes.io/projected/658d33ef-d90b-448b-b4d0-e175b77a187f-kube-api-access-mpnv5\") pod \"openshift-state-metrics-7bd7489957-pwfwf\" (UID: \"658d33ef-d90b-448b-b4d0-e175b77a187f\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-pwfwf"
Oct 31 07:28:57.118038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.118021    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7b5w\" (UniqueName: \"kubernetes.io/projected/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-kube-api-access-m7b5w\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.118544 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.118526    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2wzb\" (UniqueName: \"kubernetes.io/projected/c0fd0630-ea16-413c-a195-ff4cd58d8d21-kube-api-access-x2wzb\") pod \"alertmanager-main-0\" (UID: \"c0fd0630-ea16-413c-a195-ff4cd58d8d21\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 07:28:57.118622 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.118602    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6l87\" (UniqueName: \"kubernetes.io/projected/df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea-kube-api-access-w6l87\") pod \"monitoring-plugin-77f4f76594-cgg7t\" (UID: \"df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:28:57.118719 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.118705    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxz6l\" (UniqueName: \"kubernetes.io/projected/95b1f652-e473-4b0e-a9da-ac32940f95a8-kube-api-access-kxz6l\") pod \"kube-state-metrics-5dc8d7d89b-mrsrf\" (UID: \"95b1f652-e473-4b0e-a9da-ac32940f95a8\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-mrsrf"
Oct 31 07:28:57.118965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.118943    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffpp\" (UniqueName: \"kubernetes.io/projected/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-kube-api-access-cffpp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.202545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202468    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-var-lib-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.202545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202502    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-socket-dir-parent\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.202545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202515    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-systemd-units\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.202545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202530    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-kubernetes\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.202545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202546    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-conf\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202586    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-var-lib-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202594    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-socket-dir-parent\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202603    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-systemd-units\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202594    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-kubernetes\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202635    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-bin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202669    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-hosts-file\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202704    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-bin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202715    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7-hosts-file\") pod \"node-resolver-v59hm\" (UID: \"ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7\") " pod="openshift-dns/node-resolver-v59hm"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202726    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-sys\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202746    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-netns\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202763    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202773    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-conf\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202783    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-sys\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202799    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-netns\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202778    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-device-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202826    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-device-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202829    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202837    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202863    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-modprobe-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202870    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.202877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202880    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-hostroot\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202902    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-registration-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202912    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-hostroot\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202923    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-conf-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202940    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-var-lib-kubelet\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202957    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-netd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202973    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202992    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-cnibin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202997    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-modprobe-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.202991    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-conf-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203014    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-var-lib-kubelet\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203015    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-netd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203034    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-registration-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203038    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-node-log\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203062    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-node-log\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203044    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-cnibin\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203065    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-run\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203103    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysctl-d\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203122    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-run\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203133    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-lib-modules\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203152    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-sys-fs\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203176    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203198    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-host\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203222    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203223    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-etc-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203232    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-sys-fs\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203246    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-lib-modules\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203249    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-multus-certs\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203271    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysconfig\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203251    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-host\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203296    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-multus-certs\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203301    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-etc-openvswitch\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203319    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-sys\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203344    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-sysconfig\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203350    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-root\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203380    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-kubelet\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203393    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-root\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203406    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-systemd\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203426    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-kubelet\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203408    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-sys\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203433    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/836cf52b-b0e8-4e98-a812-21d0cebfaf63-etc-systemd\") pod \"tuned-86nzk\" (UID: \"836cf52b-b0e8-4e98-a812-21d0cebfaf63\") " pod="openshift-cluster-node-tuning-operator/tuned-86nzk"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203433    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203460    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-system-cni-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203478    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-netns\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203501    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-dbus\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203527    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-kubelet\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203549    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-run-netns\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203558    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-bin\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203574    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-dbus\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203580    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-kubelet\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203582    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-system-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203607    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-cni-bin\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203616    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-k8s-cni-cncf-io\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203641    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-system-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203642    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-ovn\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203667    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-kubelet-config\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203689    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-ovn\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.203665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203698    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-multus\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203670    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-run-k8s-cni-cncf-io\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203711    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f0f4a456-26c2-4a45-bd30-b6795d5c03cc-kubelet-config\") pod \"global-pull-secret-syncer-82457\" (UID: \"f0f4a456-26c2-4a45-bd30-b6795d5c03cc\") " pod="kube-system/global-pull-secret-syncer-82457"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203727    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203741    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-host-var-lib-cni-multus\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203745    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2af799e3-eb79-41f6-be34-411ea2f10326-host-slash\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203761    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2af799e3-eb79-41f6-be34-411ea2f10326-host-slash\") pod \"iptables-alerter-96gd4\" (UID: \"2af799e3-eb79-41f6-be34-411ea2f10326\") " pod="openshift-network-operator/iptables-alerter-96gd4"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203771    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-wtmp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203801    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203806    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203829    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cnibin\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203845    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b62296f1-d84a-4dd1-8eb0-9f64567af021-crio-socket\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203827    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee-node-exporter-wtmp\") pod \"node-exporter-hbnc6\" (UID: \"3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee\") " pod="openshift-monitoring/node-exporter-hbnc6"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203865    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-socket-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203877    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb-cnibin\") pod \"multus-additional-cni-plugins-cc6gh\" (UID: \"486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb\") " pod="openshift-multus/multus-additional-cni-plugins-cc6gh"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203888    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-slash\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203906    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-multus-cni-dir\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203918    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-log-socket\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203926    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-host-slash\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203919    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b62296f1-d84a-4dd1-8eb0-9f64567af021-crio-socket\") pod \"insights-runtime-extractor-525tp\" (UID: \"b62296f1-d84a-4dd1-8eb0-9f64567af021\") " pod="openshift-insights/insights-runtime-extractor-525tp"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203939    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203948    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-log-socket\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203968    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203981    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203991    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-etc-kubernetes\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.203938    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-socket-dir\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.204006    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-systemd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.204013    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db612775-57b2-405c-a403-8c78f173d018-etc-selinux\") pod \"aws-ebs-csi-driver-node-bcbfl\" (UID: \"db612775-57b2-405c-a403-8c78f173d018\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bcbfl"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.204026    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fc187c9-2163-4de8-afbb-77f65de777c8-run-systemd\") pod \"ovnkube-node-4g4jz\" (UID: \"4fc187c9-2163-4de8-afbb-77f65de777c8\") " pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.204030    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e3a7ead-4527-476b-9b02-c50e68e80166-etc-kubernetes\") pod \"multus-5jj86\" (UID: \"1e3a7ead-4527-476b-9b02-c50e68e80166\") " pod="openshift-multus/multus-5jj86"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.204034    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f206a32f-819d-4e61-9040-790e8c9183ec-host\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:28:57.205301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.204086    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f206a32f-819d-4e61-9040-790e8c9183ec-host\") pod \"node-ca-wbg8z\" (UID: \"f206a32f-819d-4e61-9040-790e8c9183ec\") " pod="openshift-image-registry/node-ca-wbg8z"
Oct 31 07:28:57.581414 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:57.581267    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:58.540128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:58.540098    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8cd777578-wbgnk"
Oct 31 07:28:59.073123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:59.073092    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:28:59.073378 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:59.073364    5829 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 07:28:59.768856 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:28:59.768831    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 31 07:29:00.255161 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:00.255130    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hzqlg"
Oct 31 07:29:01.310776 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:01.310744    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cgg7t"
Oct 31 07:29:02.161970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:02.161936    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pkj5d"
Oct 31 07:29:02.805510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:02.805465    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:29:06.731226 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:06.731197    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4g4jz"
Oct 31 07:29:16.509295 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:29:16.509198    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:29:19.032434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:19.032404    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b67d5b6c9-897nz"
Oct 31 07:29:28.230942 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.230908    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-78cc5444f8-24xwn"]
Oct 31 07:29:28.235563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.235543    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.251114 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.251083    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 07:29:28.251114 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.251100    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 07:29:28.251344 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.251162    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 07:29:28.251389 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.251376    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 07:29:28.264160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.264121    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t667t\""
Oct 31 07:29:28.265043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.265022    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78cc5444f8-24xwn"]
Oct 31 07:29:28.317156 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.317126    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 07:29:28.330862 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.330835    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 07:29:28.381870 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.381833    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.382042 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.381882    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-federate-client-tls\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.382042 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.381969    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-serving-certs-ca-bundle\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.382133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.382055    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-secret-telemeter-client\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.382133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.382074    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdrm\" (UniqueName: \"kubernetes.io/projected/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-kube-api-access-2jdrm\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.382133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.382095    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-telemeter-client-tls\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.382133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.382118    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-metrics-client-ca\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.382271 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.382243    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.482924 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.482836    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-serving-certs-ca-bundle\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.482924 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.482901    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-secret-telemeter-client\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.482924 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.482917    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdrm\" (UniqueName: \"kubernetes.io/projected/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-kube-api-access-2jdrm\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.483164 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.482938    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-telemeter-client-tls\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.483164 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.482961    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-metrics-client-ca\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.483377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.483187    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.483377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.483257    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.483377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.483328    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-federate-client-tls\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.483538 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.483391    5829 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 07:29:28.483829 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.483788    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-metrics-client-ca\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.484025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.484001    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-serving-certs-ca-bundle\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.484109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.484088    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.485818 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.485791    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-secret-telemeter-client\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.485941 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.485920    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-federate-client-tls\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.486035 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.486017    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-telemeter-client-tls\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.486073 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.486049    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.502390 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.502356    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdrm\" (UniqueName: \"kubernetes.io/projected/2f9af2bb-1fa9-4342-9e95-0ee6b79688e8-kube-api-access-2jdrm\") pod \"telemeter-client-78cc5444f8-24xwn\" (UID: \"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8\") " pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.548376 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.548344    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn"
Oct 31 07:29:28.873464 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.873381    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-78cc5444f8-24xwn"]
Oct 31 07:29:28.879846 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:29:28.879820    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9af2bb_1fa9_4342_9e95_0ee6b79688e8.slice/crio-708e76b6bf6b85eebf41f16824439dd23233cda28151e2dba8e4603735d7d6dd WatchSource:0}: Error finding container 708e76b6bf6b85eebf41f16824439dd23233cda28151e2dba8e4603735d7d6dd: Status 404 returned error can't find the container with id 708e76b6bf6b85eebf41f16824439dd23233cda28151e2dba8e4603735d7d6dd
Oct 31 07:29:28.881818 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:28.881802    5829 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:29:29.157802 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:29.157771    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn" event={"ID":"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8","Type":"ContainerStarted","Data":"708e76b6bf6b85eebf41f16824439dd23233cda28151e2dba8e4603735d7d6dd"}
Oct 31 07:29:31.167805 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:31.167769    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn" event={"ID":"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8","Type":"ContainerStarted","Data":"6ec9025d70487545de2748a7f711c16da0180486e14258cd0f0fe4fb7246d576"}
Oct 31 07:29:31.168204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:31.167854    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn" event={"ID":"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8","Type":"ContainerStarted","Data":"6e169c5cf91df5c2bde9d2e2363a5f7cbe572d33c92ab560914dc947180720d2"}
Oct 31 07:29:31.168204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:31.167873    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn" event={"ID":"2f9af2bb-1fa9-4342-9e95-0ee6b79688e8","Type":"ContainerStarted","Data":"7dab662e35188e70cb9d37112fb154fc7dfd091752be6fe14e4d6067649058d3"}
Oct 31 07:29:31.248470 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:31.248400    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-78cc5444f8-24xwn" podStartSLOduration=1.637571246 podStartE2EDuration="3.248385237s" totalImagesPullingTime="1.610813991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:31.234994916 +0000 UTC m=+35.374155505" watchObservedRunningTime="2026-10-31 07:29:31.248385237 +0000 UTC m=+35.387545824"
Oct 31 07:29:32.808348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:29:32.808314    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zb9td"
Oct 31 07:29:46.509869 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:29:46.509830    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:30:16.510522 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:30:16.510481    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:30:46.508721 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:30:46.508642    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:31:16.510024 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:31:16.509983    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:31:46.509541 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:31:46.509504    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:32:16.510177 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:32:16.510085    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:32:25.484164 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.484125    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"]
Oct 31 07:32:25.487050 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.487028    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.504849 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.504819    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 07:32:25.517807 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.517773    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 07:32:25.517960 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.517933    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-d6wbd\""
Oct 31 07:32:25.518598 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.518578    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"]
Oct 31 07:32:25.607762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.607668    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.607762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.607714    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.607762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.607736    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqhdf\" (UniqueName: \"kubernetes.io/projected/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-kube-api-access-sqhdf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.708757 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.708718    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.708757 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.708758    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.708967 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.708776    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqhdf\" (UniqueName: \"kubernetes.io/projected/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-kube-api-access-sqhdf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.709116 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.709094    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.709150 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.709125    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.728959 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.728921    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqhdf\" (UniqueName: \"kubernetes.io/projected/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-kube-api-access-sqhdf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:25.799826 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:25.799788    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:26.095528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:26.095489    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"]
Oct 31 07:32:26.100458 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:32:26.100419    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7b0cb6_b30c_43ce_8582_269ef4bb71e7.slice/crio-74ba8ddabecf646233be872cee764279ec04878b2bcade3c7ed2514c41f70bca WatchSource:0}: Error finding container 74ba8ddabecf646233be872cee764279ec04878b2bcade3c7ed2514c41f70bca: Status 404 returned error can't find the container with id 74ba8ddabecf646233be872cee764279ec04878b2bcade3c7ed2514c41f70bca
Oct 31 07:32:26.823289 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:26.823242    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp" event={"ID":"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7","Type":"ContainerStarted","Data":"74ba8ddabecf646233be872cee764279ec04878b2bcade3c7ed2514c41f70bca"}
Oct 31 07:32:27.816600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.816567    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"]
Oct 31 07:32:27.819157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.819130    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:27.825396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.825371    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:27.825805 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.825424    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:27.825805 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.825554    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktnd\" (UniqueName: \"kubernetes.io/projected/68ce0601-6a76-4953-96c4-d63153b12a29-kube-api-access-lktnd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:27.850533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.850497    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-5gvld\""
Oct 31 07:32:27.852378 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.852357    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"]
Oct 31 07:32:27.852480 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.852413    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"]
Oct 31 07:32:27.855310 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.855265    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:27.866624 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.866597    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"]
Oct 31 07:32:27.926686 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.926651    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:27.926883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.926706    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:27.926883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.926749    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktnd\" (UniqueName: \"kubernetes.io/projected/68ce0601-6a76-4953-96c4-d63153b12a29-kube-api-access-lktnd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:27.926883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.926788    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:27.927048 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.926953    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:27.927048 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.926989    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8khp\" (UniqueName: \"kubernetes.io/projected/c8a8412a-85ac-4030-9bee-cb069164b14d-kube-api-access-r8khp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:27.927148 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.927098    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:27.927273 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.927257    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:27.940423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.940385    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-srx6k\""
Oct 31 07:32:27.946772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:27.946747    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktnd\" (UniqueName: \"kubernetes.io/projected/68ce0601-6a76-4953-96c4-d63153b12a29-kube-api-access-lktnd\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:28.028385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.028298    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:28.028385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.028362    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:28.028641 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.028454    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8khp\" (UniqueName: \"kubernetes.io/projected/c8a8412a-85ac-4030-9bee-cb069164b14d-kube-api-access-r8khp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:28.028729 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.028709    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:28.028788 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.028771    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:28.049131 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.049092    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8khp\" (UniqueName: \"kubernetes.io/projected/c8a8412a-85ac-4030-9bee-cb069164b14d-kube-api-access-r8khp\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:28.134107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.134024    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:28.171185 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.171153    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:28.439731 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.439662    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"]
Oct 31 07:32:28.446018 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:32:28.445986    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ce0601_6a76_4953_96c4_d63153b12a29.slice/crio-bf38d1389e6edb773f7a9f04fe4e5b577b5b76261d7a40442ea3b3c0b5a0ceaa WatchSource:0}: Error finding container bf38d1389e6edb773f7a9f04fe4e5b577b5b76261d7a40442ea3b3c0b5a0ceaa: Status 404 returned error can't find the container with id bf38d1389e6edb773f7a9f04fe4e5b577b5b76261d7a40442ea3b3c0b5a0ceaa
Oct 31 07:32:28.486111 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.486080    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"]
Oct 31 07:32:28.490057 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:32:28.490024    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8a8412a_85ac_4030_9bee_cb069164b14d.slice/crio-a7764f17dfc7c5dfe2e2fa2c56e2f50089686152d67ba7325edd768e47c1f861 WatchSource:0}: Error finding container a7764f17dfc7c5dfe2e2fa2c56e2f50089686152d67ba7325edd768e47c1f861: Status 404 returned error can't find the container with id a7764f17dfc7c5dfe2e2fa2c56e2f50089686152d67ba7325edd768e47c1f861
Oct 31 07:32:28.834851 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.834731    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw" event={"ID":"c8a8412a-85ac-4030-9bee-cb069164b14d","Type":"ContainerStarted","Data":"a7764f17dfc7c5dfe2e2fa2c56e2f50089686152d67ba7325edd768e47c1f861"}
Oct 31 07:32:28.836560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:28.836525    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr" event={"ID":"68ce0601-6a76-4953-96c4-d63153b12a29","Type":"ContainerStarted","Data":"bf38d1389e6edb773f7a9f04fe4e5b577b5b76261d7a40442ea3b3c0b5a0ceaa"}
Oct 31 07:32:32.858917 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:32.858839    5829 generic.go:356] "Generic (PLEG): container finished" podID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerID="b288efa2cad454a4a24668413bdda1575892cbd885687d906c946741dcb7744f" exitCode=0
Oct 31 07:32:32.859414 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:32.858923    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp" event={"ID":"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7","Type":"ContainerDied","Data":"b288efa2cad454a4a24668413bdda1575892cbd885687d906c946741dcb7744f"}
Oct 31 07:32:32.860748 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:32.860657    5829 generic.go:356] "Generic (PLEG): container finished" podID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerID="d984265f6be8b1115a3bfca7db91151daa991128c2546a395ae3303344750daf" exitCode=0
Oct 31 07:32:32.860748 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:32.860696    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw" event={"ID":"c8a8412a-85ac-4030-9bee-cb069164b14d","Type":"ContainerDied","Data":"d984265f6be8b1115a3bfca7db91151daa991128c2546a395ae3303344750daf"}
Oct 31 07:32:33.867562 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:33.867528    5829 generic.go:356] "Generic (PLEG): container finished" podID="68ce0601-6a76-4953-96c4-d63153b12a29" containerID="2c22443e888dd9b22adc75b78cc34ec8d2f30be029442e8a0a9bef2920f0f66b" exitCode=0
Oct 31 07:32:33.868069 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:33.867636    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr" event={"ID":"68ce0601-6a76-4953-96c4-d63153b12a29","Type":"ContainerDied","Data":"2c22443e888dd9b22adc75b78cc34ec8d2f30be029442e8a0a9bef2920f0f66b"}
Oct 31 07:32:33.870053 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:33.870027    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw" event={"ID":"c8a8412a-85ac-4030-9bee-cb069164b14d","Type":"ContainerStarted","Data":"06005d0ec86006a7325ba9438d99a64fd6859dcb483d2ee105fc42457a7457a6"}
Oct 31 07:32:34.874964 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:34.874932    5829 generic.go:356] "Generic (PLEG): container finished" podID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerID="06005d0ec86006a7325ba9438d99a64fd6859dcb483d2ee105fc42457a7457a6" exitCode=0
Oct 31 07:32:34.875357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:34.875014    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw" event={"ID":"c8a8412a-85ac-4030-9bee-cb069164b14d","Type":"ContainerDied","Data":"06005d0ec86006a7325ba9438d99a64fd6859dcb483d2ee105fc42457a7457a6"}
Oct 31 07:32:34.876897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:34.876869    5829 generic.go:356] "Generic (PLEG): container finished" podID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerID="83683f53edffc473cc6876bfba8102c1e18211df3b69e847497d60685237bc19" exitCode=0
Oct 31 07:32:34.877001 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:34.876914    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp" event={"ID":"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7","Type":"ContainerDied","Data":"83683f53edffc473cc6876bfba8102c1e18211df3b69e847497d60685237bc19"}
Oct 31 07:32:34.878907 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:34.878886    5829 generic.go:356] "Generic (PLEG): container finished" podID="68ce0601-6a76-4953-96c4-d63153b12a29" containerID="7458dd279d6f3733d512bacf84537252f4a4ce4226d801d9c13e3538f077c312" exitCode=0
Oct 31 07:32:34.878962 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:34.878926    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr" event={"ID":"68ce0601-6a76-4953-96c4-d63153b12a29","Type":"ContainerDied","Data":"7458dd279d6f3733d512bacf84537252f4a4ce4226d801d9c13e3538f077c312"}
Oct 31 07:32:42.921750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:42.921660    5829 generic.go:356] "Generic (PLEG): container finished" podID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerID="ca12172c758479249f9b51a1b93cd3c40c1912d460e23f05c4247356ece42dc0" exitCode=0
Oct 31 07:32:42.922253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:42.921753    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp" event={"ID":"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7","Type":"ContainerDied","Data":"ca12172c758479249f9b51a1b93cd3c40c1912d460e23f05c4247356ece42dc0"}
Oct 31 07:32:42.923904 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:42.923885    5829 generic.go:356] "Generic (PLEG): container finished" podID="68ce0601-6a76-4953-96c4-d63153b12a29" containerID="7b3b8d39e1f5dda7f47c623bfad66965c6dc88ab60fd87d9df4ce5f16f3fa9bd" exitCode=0
Oct 31 07:32:42.924018 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:42.923957    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr" event={"ID":"68ce0601-6a76-4953-96c4-d63153b12a29","Type":"ContainerDied","Data":"7b3b8d39e1f5dda7f47c623bfad66965c6dc88ab60fd87d9df4ce5f16f3fa9bd"}
Oct 31 07:32:42.925883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:42.925862    5829 generic.go:356] "Generic (PLEG): container finished" podID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerID="259e3a2093aa17f915bfcdad96a3544b2f38f9ab3dacff1f6a2731b53238cb73" exitCode=0
Oct 31 07:32:42.926012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:42.925942    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw" event={"ID":"c8a8412a-85ac-4030-9bee-cb069164b14d","Type":"ContainerDied","Data":"259e3a2093aa17f915bfcdad96a3544b2f38f9ab3dacff1f6a2731b53238cb73"}
Oct 31 07:32:44.068338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.068314    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:44.105650 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.105626    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:44.108456 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.108433    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:44.179728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.179702    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-util\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-util\") pod \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") "
Oct 31 07:32:44.179908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.179755    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-bundle\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-bundle\") pod \"68ce0601-6a76-4953-96c4-d63153b12a29\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") "
Oct 31 07:32:44.179908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.179787    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-util\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-util\") pod \"68ce0601-6a76-4953-96c4-d63153b12a29\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") "
Oct 31 07:32:44.179908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.179804    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-bundle\") pod \"c8a8412a-85ac-4030-9bee-cb069164b14d\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") "
Oct 31 07:32:44.179908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.179829    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-kube-api-access-sqhdf\" (UniqueName: \"kubernetes.io/projected/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-kube-api-access-sqhdf\") pod \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") "
Oct 31 07:32:44.179908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.179867    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-util\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-util\") pod \"c8a8412a-85ac-4030-9bee-cb069164b14d\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") "
Oct 31 07:32:44.179908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.179896    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8a8412a-85ac-4030-9bee-cb069164b14d-kube-api-access-r8khp\" (UniqueName: \"kubernetes.io/projected/c8a8412a-85ac-4030-9bee-cb069164b14d-kube-api-access-r8khp\") pod \"c8a8412a-85ac-4030-9bee-cb069164b14d\" (UID: \"c8a8412a-85ac-4030-9bee-cb069164b14d\") "
Oct 31 07:32:44.180228 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.179963    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ce0601-6a76-4953-96c4-d63153b12a29-kube-api-access-lktnd\" (UniqueName: \"kubernetes.io/projected/68ce0601-6a76-4953-96c4-d63153b12a29-kube-api-access-lktnd\") pod \"68ce0601-6a76-4953-96c4-d63153b12a29\" (UID: \"68ce0601-6a76-4953-96c4-d63153b12a29\") "
Oct 31 07:32:44.180228 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.180016    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-bundle\") pod \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\" (UID: \"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7\") "
Oct 31 07:32:44.180762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.180681    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-bundle" pod "c8a8412a-85ac-4030-9bee-cb069164b14d" (UID: "c8a8412a-85ac-4030-9bee-cb069164b14d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:32:44.180762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.180710    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-bundle" pod "68ce0601-6a76-4953-96c4-d63153b12a29" (UID: "68ce0601-6a76-4953-96c4-d63153b12a29"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:32:44.180987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.180962    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-bundle" pod "1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" (UID: "1b7b0cb6-b30c-43ce-8582-269ef4bb71e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:32:44.182553 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.182515    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a8412a-85ac-4030-9bee-cb069164b14d-kube-api-access-r8khp" pod "c8a8412a-85ac-4030-9bee-cb069164b14d" (UID: "c8a8412a-85ac-4030-9bee-cb069164b14d"). InnerVolumeSpecName "kube-api-access-r8khp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:32:44.182668 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.182556    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-kube-api-access-sqhdf" pod "1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" (UID: "1b7b0cb6-b30c-43ce-8582-269ef4bb71e7"). InnerVolumeSpecName "kube-api-access-sqhdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:32:44.182668 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.182638    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ce0601-6a76-4953-96c4-d63153b12a29-kube-api-access-lktnd" pod "68ce0601-6a76-4953-96c4-d63153b12a29" (UID: "68ce0601-6a76-4953-96c4-d63153b12a29"). InnerVolumeSpecName "kube-api-access-lktnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:32:44.184846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.184820    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-util" pod "c8a8412a-85ac-4030-9bee-cb069164b14d" (UID: "c8a8412a-85ac-4030-9bee-cb069164b14d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:32:44.186026 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.186005    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-util" pod "68ce0601-6a76-4953-96c4-d63153b12a29" (UID: "68ce0601-6a76-4953-96c4-d63153b12a29"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:32:44.186181 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.186161    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-util" pod "1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" (UID: "1b7b0cb6-b30c-43ce-8582-269ef4bb71e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:32:44.280764 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.280728    5829 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-util\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:32:44.280764 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.280756    5829 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-bundle\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:32:44.280764 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.280765    5829 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/68ce0601-6a76-4953-96c4-d63153b12a29-util\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:32:44.280764 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.280774    5829 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-bundle\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:32:44.281014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.280784    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqhdf\" (UniqueName: \"kubernetes.io/projected/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-kube-api-access-sqhdf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:32:44.281014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.280793    5829 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c8a8412a-85ac-4030-9bee-cb069164b14d-util\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:32:44.281014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.280801    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8khp\" (UniqueName: \"kubernetes.io/projected/c8a8412a-85ac-4030-9bee-cb069164b14d-kube-api-access-r8khp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:32:44.281014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.280809    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lktnd\" (UniqueName: \"kubernetes.io/projected/68ce0601-6a76-4953-96c4-d63153b12a29-kube-api-access-lktnd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:32:44.281014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.280817    5829 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1b7b0cb6-b30c-43ce-8582-269ef4bb71e7-bundle\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:32:44.935563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.935533    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw"
Oct 31 07:32:44.935563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.935554    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tbccw" event={"ID":"c8a8412a-85ac-4030-9bee-cb069164b14d","Type":"ContainerDied","Data":"a7764f17dfc7c5dfe2e2fa2c56e2f50089686152d67ba7325edd768e47c1f861"}
Oct 31 07:32:44.935803 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.935609    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7764f17dfc7c5dfe2e2fa2c56e2f50089686152d67ba7325edd768e47c1f861"
Oct 31 07:32:44.937632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.937609    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp" event={"ID":"1b7b0cb6-b30c-43ce-8582-269ef4bb71e7","Type":"ContainerDied","Data":"74ba8ddabecf646233be872cee764279ec04878b2bcade3c7ed2514c41f70bca"}
Oct 31 07:32:44.937632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.937621    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmsrlp"
Oct 31 07:32:44.937811 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.937643    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74ba8ddabecf646233be872cee764279ec04878b2bcade3c7ed2514c41f70bca"
Oct 31 07:32:44.945850 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.945818    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr" event={"ID":"68ce0601-6a76-4953-96c4-d63153b12a29","Type":"ContainerDied","Data":"bf38d1389e6edb773f7a9f04fe4e5b577b5b76261d7a40442ea3b3c0b5a0ceaa"}
Oct 31 07:32:44.945985 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.945875    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6w54lr"
Oct 31 07:32:44.945985 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:44.945883    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf38d1389e6edb773f7a9f04fe4e5b577b5b76261d7a40442ea3b3c0b5a0ceaa"
Oct 31 07:32:46.510533 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:32:46.510495    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:32:48.522516 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.522478    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"]
Oct 31 07:32:48.523028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.522941    5829 cpu_manager.go:414] "containerMap: removing container" podUID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerName="util"
Oct 31 07:32:48.523028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.522960    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerName="util"
Oct 31 07:32:48.523028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.522973    5829 cpu_manager.go:414] "containerMap: removing container" podUID="68ce0601-6a76-4953-96c4-d63153b12a29" containerName="pull"
Oct 31 07:32:48.523028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.522980    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ce0601-6a76-4953-96c4-d63153b12a29" containerName="pull"
Oct 31 07:32:48.523028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.522996    5829 cpu_manager.go:414] "containerMap: removing container" podUID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerName="extract"
Oct 31 07:32:48.523028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523004    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerName="extract"
Oct 31 07:32:48.523028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523015    5829 cpu_manager.go:414] "containerMap: removing container" podUID="68ce0601-6a76-4953-96c4-d63153b12a29" containerName="extract"
Oct 31 07:32:48.523028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523021    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ce0601-6a76-4953-96c4-d63153b12a29" containerName="extract"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523042    5829 cpu_manager.go:414] "containerMap: removing container" podUID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerName="util"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523049    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerName="util"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523062    5829 cpu_manager.go:414] "containerMap: removing container" podUID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerName="pull"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523069    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerName="pull"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523079    5829 cpu_manager.go:414] "containerMap: removing container" podUID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerName="extract"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523086    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerName="extract"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523109    5829 cpu_manager.go:414] "containerMap: removing container" podUID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerName="pull"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523115    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerName="pull"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523133    5829 cpu_manager.go:414] "containerMap: removing container" podUID="68ce0601-6a76-4953-96c4-d63153b12a29" containerName="util"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523141    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ce0601-6a76-4953-96c4-d63153b12a29" containerName="util"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523218    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7b0cb6-b30c-43ce-8582-269ef4bb71e7" containerName="extract"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523235    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8a8412a-85ac-4030-9bee-cb069164b14d" containerName="extract"
Oct 31 07:32:48.523657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.523244    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ce0601-6a76-4953-96c4-d63153b12a29" containerName="extract"
Oct 31 07:32:48.526561 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.526539    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"
Oct 31 07:32:48.538721 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.538690    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"]
Oct 31 07:32:48.550917 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.550890    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:32:48.550917 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.550890    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 07:32:48.551099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.550939    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rz6mj\""
Oct 31 07:32:48.616467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.616423    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945565d4-4476-49c4-96ad-f1ce44144804-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7\" (UID: \"945565d4-4476-49c4-96ad-f1ce44144804\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"
Oct 31 07:32:48.616648 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.616476    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlnx\" (UniqueName: \"kubernetes.io/projected/945565d4-4476-49c4-96ad-f1ce44144804-kube-api-access-khlnx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7\" (UID: \"945565d4-4476-49c4-96ad-f1ce44144804\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"
Oct 31 07:32:48.717579 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.717541    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlnx\" (UniqueName: \"kubernetes.io/projected/945565d4-4476-49c4-96ad-f1ce44144804-kube-api-access-khlnx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7\" (UID: \"945565d4-4476-49c4-96ad-f1ce44144804\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"
Oct 31 07:32:48.717754 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.717643    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945565d4-4476-49c4-96ad-f1ce44144804-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7\" (UID: \"945565d4-4476-49c4-96ad-f1ce44144804\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"
Oct 31 07:32:48.718001 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.717983    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945565d4-4476-49c4-96ad-f1ce44144804-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7\" (UID: \"945565d4-4476-49c4-96ad-f1ce44144804\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"
Oct 31 07:32:48.743992 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.743958    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlnx\" (UniqueName: \"kubernetes.io/projected/945565d4-4476-49c4-96ad-f1ce44144804-kube-api-access-khlnx\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7\" (UID: \"945565d4-4476-49c4-96ad-f1ce44144804\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"
Oct 31 07:32:48.838793 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:48.838704    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"
Oct 31 07:32:49.167664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:49.167635    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7"]
Oct 31 07:32:49.185803 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:32:49.185774    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945565d4_4476_49c4_96ad_f1ce44144804.slice/crio-79db6831c155d019d2f778e099de02ea6977eb0a0e7444270df3048a855df7a1 WatchSource:0}: Error finding container 79db6831c155d019d2f778e099de02ea6977eb0a0e7444270df3048a855df7a1: Status 404 returned error can't find the container with id 79db6831c155d019d2f778e099de02ea6977eb0a0e7444270df3048a855df7a1
Oct 31 07:32:49.971915 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:49.971879    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7" event={"ID":"945565d4-4476-49c4-96ad-f1ce44144804","Type":"ContainerStarted","Data":"79db6831c155d019d2f778e099de02ea6977eb0a0e7444270df3048a855df7a1"}
Oct 31 07:32:50.974874 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:50.974824    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7" event={"ID":"945565d4-4476-49c4-96ad-f1ce44144804","Type":"ContainerStarted","Data":"d1ea9b378d7ac5ad670c49ef1a85397960d4de3b57ae9b865753728234cbc5f4"}
Oct 31 07:32:51.043908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:51.043855    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-hk4s7" podStartSLOduration=1.3764149909999999 podStartE2EDuration="3.043840855s" totalImagesPullingTime="1.667425864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:32:51.030301156 +0000 UTC m=+235.169461741" watchObservedRunningTime="2026-10-31 07:32:51.043840855 +0000 UTC m=+235.183001443"
Oct 31 07:32:54.536923 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.536891    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"]
Oct 31 07:32:54.540381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.540362    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"
Oct 31 07:32:54.559452 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.559426    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"]
Oct 31 07:32:54.571267 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.571241    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 07:32:54.571267 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.571241    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 07:32:54.571267 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.571251    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vrqtd\""
Oct 31 07:32:54.674743 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.674708    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba89e7ed-4e43-4be9-b416-6ef6e930fbea-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qxp8q\" (UID: \"ba89e7ed-4e43-4be9-b416-6ef6e930fbea\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"
Oct 31 07:32:54.674743 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.674752    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss568\" (UniqueName: \"kubernetes.io/projected/ba89e7ed-4e43-4be9-b416-6ef6e930fbea-kube-api-access-ss568\") pod \"cert-manager-cainjector-5c57bbd4fc-qxp8q\" (UID: \"ba89e7ed-4e43-4be9-b416-6ef6e930fbea\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"
Oct 31 07:32:54.775883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.775850    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba89e7ed-4e43-4be9-b416-6ef6e930fbea-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qxp8q\" (UID: \"ba89e7ed-4e43-4be9-b416-6ef6e930fbea\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"
Oct 31 07:32:54.776082 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.775888    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss568\" (UniqueName: \"kubernetes.io/projected/ba89e7ed-4e43-4be9-b416-6ef6e930fbea-kube-api-access-ss568\") pod \"cert-manager-cainjector-5c57bbd4fc-qxp8q\" (UID: \"ba89e7ed-4e43-4be9-b416-6ef6e930fbea\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"
Oct 31 07:32:54.796369 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.796275    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba89e7ed-4e43-4be9-b416-6ef6e930fbea-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-qxp8q\" (UID: \"ba89e7ed-4e43-4be9-b416-6ef6e930fbea\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"
Oct 31 07:32:54.796503 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.796483    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss568\" (UniqueName: \"kubernetes.io/projected/ba89e7ed-4e43-4be9-b416-6ef6e930fbea-kube-api-access-ss568\") pod \"cert-manager-cainjector-5c57bbd4fc-qxp8q\" (UID: \"ba89e7ed-4e43-4be9-b416-6ef6e930fbea\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"
Oct 31 07:32:54.862404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:54.862365    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"
Oct 31 07:32:55.158545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:55.158511    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q"]
Oct 31 07:32:55.165501 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:32:55.165461    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba89e7ed_4e43_4be9_b416_6ef6e930fbea.slice/crio-884eb77cc01a3a1815b86ad6e216a9b72a3e566a13c7172e11576509cee32084 WatchSource:0}: Error finding container 884eb77cc01a3a1815b86ad6e216a9b72a3e566a13c7172e11576509cee32084: Status 404 returned error can't find the container with id 884eb77cc01a3a1815b86ad6e216a9b72a3e566a13c7172e11576509cee32084
Oct 31 07:32:55.997364 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:55.997298    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q" event={"ID":"ba89e7ed-4e43-4be9-b416-6ef6e930fbea","Type":"ContainerStarted","Data":"884eb77cc01a3a1815b86ad6e216a9b72a3e566a13c7172e11576509cee32084"}
Oct 31 07:32:58.009018 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:58.008974    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q" event={"ID":"ba89e7ed-4e43-4be9-b416-6ef6e930fbea","Type":"ContainerStarted","Data":"ca7b13b56117e21a15595c0ee794ccbd4deab6655e5fcd7c03ae41bd15cb0b8d"}
Oct 31 07:32:58.048485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:58.048376    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-qxp8q" podStartSLOduration=1.4243484149999999 podStartE2EDuration="4.048354518s" totalImagesPullingTime="2.624006103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:32:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:32:58.047606176 +0000 UTC m=+242.186766765" watchObservedRunningTime="2026-10-31 07:32:58.048354518 +0000 UTC m=+242.187515112"
Oct 31 07:32:59.391297 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.391260    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"]
Oct 31 07:32:59.394686 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.394670    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:32:59.411645 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.411619    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"]
Oct 31 07:32:59.423591 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.423561    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-brgxg\""
Oct 31 07:32:59.516689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.516642    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2khk6\" (UID: \"67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:32:59.516689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.516684    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwwnb\" (UniqueName: \"kubernetes.io/projected/67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe-kube-api-access-bwwnb\") pod \"cert-manager-webhook-5dbbbf75cc-2khk6\" (UID: \"67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:32:59.617890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.617854    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2khk6\" (UID: \"67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:32:59.617890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.617894    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwwnb\" (UniqueName: \"kubernetes.io/projected/67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe-kube-api-access-bwwnb\") pod \"cert-manager-webhook-5dbbbf75cc-2khk6\" (UID: \"67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:32:59.638057 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.638031    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2khk6\" (UID: \"67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:32:59.638249 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.638226    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwwnb\" (UniqueName: \"kubernetes.io/projected/67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe-kube-api-access-bwwnb\") pod \"cert-manager-webhook-5dbbbf75cc-2khk6\" (UID: \"67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:32:59.709393 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:32:59.709295    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:33:00.023864 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:00.023753    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"]
Oct 31 07:33:00.027638 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:33:00.027603    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67ce1496_b71b_4aca_a7f5_ac8dbd09b5fe.slice/crio-9ec5b1844b29e39caf3e11730e6ba470850d04dc9fd35bedce4bcff872ef6fc9 WatchSource:0}: Error finding container 9ec5b1844b29e39caf3e11730e6ba470850d04dc9fd35bedce4bcff872ef6fc9: Status 404 returned error can't find the container with id 9ec5b1844b29e39caf3e11730e6ba470850d04dc9fd35bedce4bcff872ef6fc9
Oct 31 07:33:01.023512 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:01.023470    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6" event={"ID":"67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe","Type":"ContainerStarted","Data":"051bfdd56b8f17a4dcc3d42e8d0cec47cfbcfeceabae89ee047879707ee1e5c8"}
Oct 31 07:33:01.023907 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:01.023524    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6" event={"ID":"67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe","Type":"ContainerStarted","Data":"9ec5b1844b29e39caf3e11730e6ba470850d04dc9fd35bedce4bcff872ef6fc9"}
Oct 31 07:33:01.023907 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:01.023650    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:33:01.063482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:01.063423    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6" podStartSLOduration=2.063407896 podStartE2EDuration="2.063407896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:33:01.062352268 +0000 UTC m=+245.201512855" watchObservedRunningTime="2026-10-31 07:33:01.063407896 +0000 UTC m=+245.202568483"
Oct 31 07:33:05.124783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.124754    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"]
Oct 31 07:33:05.130437 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.130405    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"
Oct 31 07:33:05.146649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.146618    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"]
Oct 31 07:33:05.158715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.158680    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-fxmlh\""
Oct 31 07:33:05.158715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.158703    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 07:33:05.158715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.158679    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 07:33:05.267916 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.267874    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a491400b-ac46-4ef2-8361-ef6694f91c14-tmp\") pod \"jobset-operator-674cd59dc5-4wm9m\" (UID: \"a491400b-ac46-4ef2-8361-ef6694f91c14\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"
Oct 31 07:33:05.268098 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.267937    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b752l\" (UniqueName: \"kubernetes.io/projected/a491400b-ac46-4ef2-8361-ef6694f91c14-kube-api-access-b752l\") pod \"jobset-operator-674cd59dc5-4wm9m\" (UID: \"a491400b-ac46-4ef2-8361-ef6694f91c14\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"
Oct 31 07:33:05.368747 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.368712    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a491400b-ac46-4ef2-8361-ef6694f91c14-tmp\") pod \"jobset-operator-674cd59dc5-4wm9m\" (UID: \"a491400b-ac46-4ef2-8361-ef6694f91c14\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"
Oct 31 07:33:05.368914 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.368759    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b752l\" (UniqueName: \"kubernetes.io/projected/a491400b-ac46-4ef2-8361-ef6694f91c14-kube-api-access-b752l\") pod \"jobset-operator-674cd59dc5-4wm9m\" (UID: \"a491400b-ac46-4ef2-8361-ef6694f91c14\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"
Oct 31 07:33:05.369127 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.369106    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a491400b-ac46-4ef2-8361-ef6694f91c14-tmp\") pod \"jobset-operator-674cd59dc5-4wm9m\" (UID: \"a491400b-ac46-4ef2-8361-ef6694f91c14\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"
Oct 31 07:33:05.388636 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.388601    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b752l\" (UniqueName: \"kubernetes.io/projected/a491400b-ac46-4ef2-8361-ef6694f91c14-kube-api-access-b752l\") pod \"jobset-operator-674cd59dc5-4wm9m\" (UID: \"a491400b-ac46-4ef2-8361-ef6694f91c14\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"
Oct 31 07:33:05.441882 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.441831    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"
Oct 31 07:33:05.738169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:05.738072    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m"]
Oct 31 07:33:05.747072 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:33:05.747037    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda491400b_ac46_4ef2_8361_ef6694f91c14.slice/crio-cec9148659bb88de59c1a56007ec328b9bede19251b6e229cf0dea5daa1e4ed9 WatchSource:0}: Error finding container cec9148659bb88de59c1a56007ec328b9bede19251b6e229cf0dea5daa1e4ed9: Status 404 returned error can't find the container with id cec9148659bb88de59c1a56007ec328b9bede19251b6e229cf0dea5daa1e4ed9
Oct 31 07:33:06.048976 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:06.048890    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m" event={"ID":"a491400b-ac46-4ef2-8361-ef6694f91c14","Type":"ContainerStarted","Data":"cec9148659bb88de59c1a56007ec328b9bede19251b6e229cf0dea5daa1e4ed9"}
Oct 31 07:33:07.031392 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:07.030905    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6"
Oct 31 07:33:09.064607 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:09.064570    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m" event={"ID":"a491400b-ac46-4ef2-8361-ef6694f91c14","Type":"ContainerStarted","Data":"d540bf3d54d6623b3b581147957e717501e372c62c8181c45a41d1b0f87c045a"}
Oct 31 07:33:09.104500 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:09.104448    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-4wm9m" podStartSLOduration=1.397465795 podStartE2EDuration="4.104428278s" totalImagesPullingTime="2.706962483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:33:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:33:09.103406267 +0000 UTC m=+253.242566854" watchObservedRunningTime="2026-10-31 07:33:09.104428278 +0000 UTC m=+253.243588870"
Oct 31 07:33:16.510488 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:33:16.510445    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:33:29.109363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.109330    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"]
Oct 31 07:33:29.115119 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.115101    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.147991 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.147953    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 07:33:29.147991 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.147979    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 07:33:29.147991 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.147970    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 07:33:29.148258 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.148039    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 07:33:29.148258 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.148136    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jrdcs\""
Oct 31 07:33:29.149036 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.149015    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"]
Oct 31 07:33:29.173239 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.173202    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/565e1131-e783-4a77-8b24-71d6349f545e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-kdptv\" (UID: \"565e1131-e783-4a77-8b24-71d6349f545e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.173430 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.173352    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/565e1131-e783-4a77-8b24-71d6349f545e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-kdptv\" (UID: \"565e1131-e783-4a77-8b24-71d6349f545e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.173430 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.173385    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28bsz\" (UniqueName: \"kubernetes.io/projected/565e1131-e783-4a77-8b24-71d6349f545e-kube-api-access-28bsz\") pod \"kuadrant-console-plugin-66c4fb575d-kdptv\" (UID: \"565e1131-e783-4a77-8b24-71d6349f545e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.274621 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.274579    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/565e1131-e783-4a77-8b24-71d6349f545e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-kdptv\" (UID: \"565e1131-e783-4a77-8b24-71d6349f545e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.274800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.274634    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28bsz\" (UniqueName: \"kubernetes.io/projected/565e1131-e783-4a77-8b24-71d6349f545e-kube-api-access-28bsz\") pod \"kuadrant-console-plugin-66c4fb575d-kdptv\" (UID: \"565e1131-e783-4a77-8b24-71d6349f545e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.274800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.274669    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/565e1131-e783-4a77-8b24-71d6349f545e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-kdptv\" (UID: \"565e1131-e783-4a77-8b24-71d6349f545e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.275265 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.275240    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/565e1131-e783-4a77-8b24-71d6349f545e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-kdptv\" (UID: \"565e1131-e783-4a77-8b24-71d6349f545e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.277228 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.277205    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/565e1131-e783-4a77-8b24-71d6349f545e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-kdptv\" (UID: \"565e1131-e783-4a77-8b24-71d6349f545e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.296388 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.296359    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28bsz\" (UniqueName: \"kubernetes.io/projected/565e1131-e783-4a77-8b24-71d6349f545e-kube-api-access-28bsz\") pod \"kuadrant-console-plugin-66c4fb575d-kdptv\" (UID: \"565e1131-e783-4a77-8b24-71d6349f545e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.427496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.427456    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"
Oct 31 07:33:29.734913 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:29.734815    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv"]
Oct 31 07:33:29.740589 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:33:29.740540    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod565e1131_e783_4a77_8b24_71d6349f545e.slice/crio-85f81b819fd9109ed6b1afd4eeff17643d76e134c29c851afcf2599770b57087 WatchSource:0}: Error finding container 85f81b819fd9109ed6b1afd4eeff17643d76e134c29c851afcf2599770b57087: Status 404 returned error can't find the container with id 85f81b819fd9109ed6b1afd4eeff17643d76e134c29c851afcf2599770b57087
Oct 31 07:33:30.162976 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:30.162940    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv" event={"ID":"565e1131-e783-4a77-8b24-71d6349f545e","Type":"ContainerStarted","Data":"85f81b819fd9109ed6b1afd4eeff17643d76e134c29c851afcf2599770b57087"}
Oct 31 07:33:30.954455 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:30.954410    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"]
Oct 31 07:33:30.958824 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:30.958796    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:30.994298 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:30.994249    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/00c31fe9-8c16-45ad-aa95-01df6db8843d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:30.994489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:30.994458    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00c31fe9-8c16-45ad-aa95-01df6db8843d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:30.994564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:30.994504    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/00c31fe9-8c16-45ad-aa95-01df6db8843d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:30.994564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:30.994536    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvhf\" (UniqueName: \"kubernetes.io/projected/00c31fe9-8c16-45ad-aa95-01df6db8843d-kube-api-access-2mvhf\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:30.994676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:30.994611    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c31fe9-8c16-45ad-aa95-01df6db8843d-tmp\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:30.995648 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:30.995361    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/00c31fe9-8c16-45ad-aa95-01df6db8843d-manifests\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.002914 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.002883    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"]
Oct 31 07:33:31.044724 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.044694    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 07:33:31.044899 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.044846    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 07:33:31.044962 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.044920    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 07:33:31.044962 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.044944    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 07:33:31.045087 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.045071    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 07:33:31.045152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.045113    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4qjz7\""
Oct 31 07:33:31.096833 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.096792    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c31fe9-8c16-45ad-aa95-01df6db8843d-tmp\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.097031 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.096845    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/00c31fe9-8c16-45ad-aa95-01df6db8843d-manifests\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.097031 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.096885    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/00c31fe9-8c16-45ad-aa95-01df6db8843d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.097031 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.096960    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00c31fe9-8c16-45ad-aa95-01df6db8843d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.097031 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.096990    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/00c31fe9-8c16-45ad-aa95-01df6db8843d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.097207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.097037    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvhf\" (UniqueName: \"kubernetes.io/projected/00c31fe9-8c16-45ad-aa95-01df6db8843d-kube-api-access-2mvhf\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.097415 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.097388    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/00c31fe9-8c16-45ad-aa95-01df6db8843d-manifests\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.099901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.099874    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00c31fe9-8c16-45ad-aa95-01df6db8843d-tmp\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.100166 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.100135    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/00c31fe9-8c16-45ad-aa95-01df6db8843d-metrics-cert\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.100311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.100171    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/00c31fe9-8c16-45ad-aa95-01df6db8843d-webhook-cert\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.100392 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.100357    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/00c31fe9-8c16-45ad-aa95-01df6db8843d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.117550 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.117520    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvhf\" (UniqueName: \"kubernetes.io/projected/00c31fe9-8c16-45ad-aa95-01df6db8843d-kube-api-access-2mvhf\") pod \"opendatahub-operator-controller-manager-6476fc869f-qhhfq\" (UID: \"00c31fe9-8c16-45ad-aa95-01df6db8843d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.280191 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.280108    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:31.655368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:31.655337    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"]
Oct 31 07:33:31.657380 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:33:31.657357    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c31fe9_8c16_45ad_aa95_01df6db8843d.slice/crio-9e16ef84873605993810843ff1c60f81bc8d7ba9b618507f2409ceafba6850d2 WatchSource:0}: Error finding container 9e16ef84873605993810843ff1c60f81bc8d7ba9b618507f2409ceafba6850d2: Status 404 returned error can't find the container with id 9e16ef84873605993810843ff1c60f81bc8d7ba9b618507f2409ceafba6850d2
Oct 31 07:33:32.175763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:32.175713    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq" event={"ID":"00c31fe9-8c16-45ad-aa95-01df6db8843d","Type":"ContainerStarted","Data":"9e16ef84873605993810843ff1c60f81bc8d7ba9b618507f2409ceafba6850d2"}
Oct 31 07:33:35.194151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:35.194113    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq" event={"ID":"00c31fe9-8c16-45ad-aa95-01df6db8843d","Type":"ContainerStarted","Data":"4b29c77e5c9d0407b9c3dd07c869887fd2f0052a859fbc2b2b07bb34541d6bf6"}
Oct 31 07:33:36.199855 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:36.199818    5829 generic.go:356] "Generic (PLEG): container finished" podID="00c31fe9-8c16-45ad-aa95-01df6db8843d" containerID="4b29c77e5c9d0407b9c3dd07c869887fd2f0052a859fbc2b2b07bb34541d6bf6" exitCode=0
Oct 31 07:33:36.200323 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:36.199910    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq" event={"ID":"00c31fe9-8c16-45ad-aa95-01df6db8843d","Type":"ContainerDied","Data":"4b29c77e5c9d0407b9c3dd07c869887fd2f0052a859fbc2b2b07bb34541d6bf6"}
Oct 31 07:33:36.201492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:36.201469    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv" event={"ID":"565e1131-e783-4a77-8b24-71d6349f545e","Type":"ContainerStarted","Data":"07f2cb90110111662d5ad1c23a67450253376f303a79d25a7494596ba4cd6199"}
Oct 31 07:33:36.307602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:36.307550    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-kdptv" podStartSLOduration=1.661522714 podStartE2EDuration="7.307536141s" totalImagesPullingTime="5.646013427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:33:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:33:36.307228021 +0000 UTC m=+280.446388609" watchObservedRunningTime="2026-10-31 07:33:36.307536141 +0000 UTC m=+280.446696766"
Oct 31 07:33:37.208065 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:37.208023    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq" event={"ID":"00c31fe9-8c16-45ad-aa95-01df6db8843d","Type":"ContainerStarted","Data":"730647bb3ec3d8d9513bb338114fa9597c111844f8dde9afa82d0dc0c6f1e93b"}
Oct 31 07:33:37.208690 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:37.208208    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:37.301093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:37.301032    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq" podStartSLOduration=3.208148066 podStartE2EDuration="7.301019433s" totalImagesPullingTime="3.505984973s" totalInitContainerRuntime="586.886394ms" isStatefulPod=false podCreationTimestamp="2026-10-31 07:33:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:33:37.2874189 +0000 UTC m=+281.426579488" watchObservedRunningTime="2026-10-31 07:33:37.301019433 +0000 UTC m=+281.440180021"
Oct 31 07:33:46.510817 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:33:46.510737    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:33:48.214936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:48.214899    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6476fc869f-qhhfq"
Oct 31 07:33:56.015780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:56.015750    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:33:56.016255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:33:56.016031    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:34:12.094975 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.094939    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"]
Oct 31 07:34:12.106682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.106659    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.117560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.117530    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"]
Oct 31 07:34:12.123704 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.123679    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 07:34:12.128780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.128701    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 07:34:12.165949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.165918    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 07:34:12.166135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.165988    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 07:34:12.166135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.166060    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f8jfd\""
Oct 31 07:34:12.181503 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.181479    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3992d138-a7d9-42f6-b3b7-6dc9e2cdb478-tmp\") pod \"kube-auth-proxy-7cdbcb4849-t7xbb\" (UID: \"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478\") " pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.181631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.181510    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpdnn\" (UniqueName: \"kubernetes.io/projected/3992d138-a7d9-42f6-b3b7-6dc9e2cdb478-kube-api-access-dpdnn\") pod \"kube-auth-proxy-7cdbcb4849-t7xbb\" (UID: \"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478\") " pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.181691 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.181670    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3992d138-a7d9-42f6-b3b7-6dc9e2cdb478-tls-certs\") pod \"kube-auth-proxy-7cdbcb4849-t7xbb\" (UID: \"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478\") " pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.282494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.282452    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpdnn\" (UniqueName: \"kubernetes.io/projected/3992d138-a7d9-42f6-b3b7-6dc9e2cdb478-kube-api-access-dpdnn\") pod \"kube-auth-proxy-7cdbcb4849-t7xbb\" (UID: \"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478\") " pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.282700 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.282610    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3992d138-a7d9-42f6-b3b7-6dc9e2cdb478-tls-certs\") pod \"kube-auth-proxy-7cdbcb4849-t7xbb\" (UID: \"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478\") " pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.282700 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.282676    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3992d138-a7d9-42f6-b3b7-6dc9e2cdb478-tmp\") pod \"kube-auth-proxy-7cdbcb4849-t7xbb\" (UID: \"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478\") " pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.285659 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.285624    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3992d138-a7d9-42f6-b3b7-6dc9e2cdb478-tmp\") pod \"kube-auth-proxy-7cdbcb4849-t7xbb\" (UID: \"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478\") " pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.285784 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.285724    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3992d138-a7d9-42f6-b3b7-6dc9e2cdb478-tls-certs\") pod \"kube-auth-proxy-7cdbcb4849-t7xbb\" (UID: \"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478\") " pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.302316 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.302266    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpdnn\" (UniqueName: \"kubernetes.io/projected/3992d138-a7d9-42f6-b3b7-6dc9e2cdb478-kube-api-access-dpdnn\") pod \"kube-auth-proxy-7cdbcb4849-t7xbb\" (UID: \"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478\") " pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.419728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.419692    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"
Oct 31 07:34:12.727113 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:12.727028    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb"]
Oct 31 07:34:12.731358 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:34:12.731328    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3992d138_a7d9_42f6_b3b7_6dc9e2cdb478.slice/crio-96f22768051dad167b7db8eb63ef795963855c0eed1fea7ce83672b26816e7c7 WatchSource:0}: Error finding container 96f22768051dad167b7db8eb63ef795963855c0eed1fea7ce83672b26816e7c7: Status 404 returned error can't find the container with id 96f22768051dad167b7db8eb63ef795963855c0eed1fea7ce83672b26816e7c7
Oct 31 07:34:13.378064 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:13.378025    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb" event={"ID":"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478","Type":"ContainerStarted","Data":"96f22768051dad167b7db8eb63ef795963855c0eed1fea7ce83672b26816e7c7"}
Oct 31 07:34:13.953793 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:13.953762    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:34:13.957554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:13.957533    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:13.974445 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:13.974409    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 07:34:13.987340 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:13.987309    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:34:13.998139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:13.998106    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf4jb\" (UniqueName: \"kubernetes.io/projected/ce64b388-ff7c-4cc1-8bea-74e05c0cfd97-kube-api-access-kf4jb\") pod \"limitador-limitador-7c9b87d574-p6h7j\" (UID: \"ce64b388-ff7c-4cc1-8bea-74e05c0cfd97\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:13.998528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:13.998254    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce64b388-ff7c-4cc1-8bea-74e05c0cfd97-config-file\") pod \"limitador-limitador-7c9b87d574-p6h7j\" (UID: \"ce64b388-ff7c-4cc1-8bea-74e05c0cfd97\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:14.049127 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.049090    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:34:14.099292 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.099258    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf4jb\" (UniqueName: \"kubernetes.io/projected/ce64b388-ff7c-4cc1-8bea-74e05c0cfd97-kube-api-access-kf4jb\") pod \"limitador-limitador-7c9b87d574-p6h7j\" (UID: \"ce64b388-ff7c-4cc1-8bea-74e05c0cfd97\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:14.099475 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.099367    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce64b388-ff7c-4cc1-8bea-74e05c0cfd97-config-file\") pod \"limitador-limitador-7c9b87d574-p6h7j\" (UID: \"ce64b388-ff7c-4cc1-8bea-74e05c0cfd97\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:14.100204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.100173    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ce64b388-ff7c-4cc1-8bea-74e05c0cfd97-config-file\") pod \"limitador-limitador-7c9b87d574-p6h7j\" (UID: \"ce64b388-ff7c-4cc1-8bea-74e05c0cfd97\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:14.122407 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.122377    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf4jb\" (UniqueName: \"kubernetes.io/projected/ce64b388-ff7c-4cc1-8bea-74e05c0cfd97-kube-api-access-kf4jb\") pod \"limitador-limitador-7c9b87d574-p6h7j\" (UID: \"ce64b388-ff7c-4cc1-8bea-74e05c0cfd97\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:14.276421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.276340    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:14.584392 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.584317    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:34:14.591499 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:34:14.591458    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce64b388_ff7c_4cc1_8bea_74e05c0cfd97.slice/crio-b5f529e84b43b7c2f19b929920b8f8a43c87a85ac41ee2059dc2978de91d9e60 WatchSource:0}: Error finding container b5f529e84b43b7c2f19b929920b8f8a43c87a85ac41ee2059dc2978de91d9e60: Status 404 returned error can't find the container with id b5f529e84b43b7c2f19b929920b8f8a43c87a85ac41ee2059dc2978de91d9e60
Oct 31 07:34:14.689898 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.689860    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"]
Oct 31 07:34:14.707853 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.706691    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.715327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.715273    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"]
Oct 31 07:34:14.725331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.722904    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 07:34:14.755037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.755008    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-lp6mz\""
Oct 31 07:34:14.755216 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.755124    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 07:34:14.755216 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.755193    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 07:34:14.806530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.806494    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d3af07-65ef-4ee0-b640-f0e8c751440a-metrics-certs\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.806707 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.806543    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60d3af07-65ef-4ee0-b640-f0e8c751440a-manager-config\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.806750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.806692    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjqx\" (UniqueName: \"kubernetes.io/projected/60d3af07-65ef-4ee0-b640-f0e8c751440a-kube-api-access-kcjqx\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.806750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.806740    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60d3af07-65ef-4ee0-b640-f0e8c751440a-cert\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.907747 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.907716    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d3af07-65ef-4ee0-b640-f0e8c751440a-metrics-certs\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.907747 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.907752    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60d3af07-65ef-4ee0-b640-f0e8c751440a-manager-config\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.907989 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.907787    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjqx\" (UniqueName: \"kubernetes.io/projected/60d3af07-65ef-4ee0-b640-f0e8c751440a-kube-api-access-kcjqx\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.907989 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.907819    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60d3af07-65ef-4ee0-b640-f0e8c751440a-cert\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.909861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.909831    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60d3af07-65ef-4ee0-b640-f0e8c751440a-manager-config\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.911512 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.911483    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60d3af07-65ef-4ee0-b640-f0e8c751440a-cert\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.911512 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.911509    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60d3af07-65ef-4ee0-b640-f0e8c751440a-metrics-certs\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:14.928382 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:14.928345    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjqx\" (UniqueName: \"kubernetes.io/projected/60d3af07-65ef-4ee0-b640-f0e8c751440a-kube-api-access-kcjqx\") pod \"jobset-controller-manager-5654d67679-dw56s\" (UID: \"60d3af07-65ef-4ee0-b640-f0e8c751440a\") " pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:15.027864 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:15.027825    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:15.335109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:15.335074    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"]
Oct 31 07:34:15.338555 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:34:15.338528    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d3af07_65ef_4ee0_b640_f0e8c751440a.slice/crio-277578d17ee5cd661c5ce88252ddc356b6f442de510e4775eef304f587de7518 WatchSource:0}: Error finding container 277578d17ee5cd661c5ce88252ddc356b6f442de510e4775eef304f587de7518: Status 404 returned error can't find the container with id 277578d17ee5cd661c5ce88252ddc356b6f442de510e4775eef304f587de7518
Oct 31 07:34:15.390318 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:15.390267    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s" event={"ID":"60d3af07-65ef-4ee0-b640-f0e8c751440a","Type":"ContainerStarted","Data":"277578d17ee5cd661c5ce88252ddc356b6f442de510e4775eef304f587de7518"}
Oct 31 07:34:15.393666 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:15.393640    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j" event={"ID":"ce64b388-ff7c-4cc1-8bea-74e05c0cfd97","Type":"ContainerStarted","Data":"b5f529e84b43b7c2f19b929920b8f8a43c87a85ac41ee2059dc2978de91d9e60"}
Oct 31 07:34:15.431263 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:15.431197    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb" podStartSLOduration=0.874229521 podStartE2EDuration="3.431182332s" totalImagesPullingTime="2.556952811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:34:15.430876564 +0000 UTC m=+319.570037152" watchObservedRunningTime="2026-10-31 07:34:15.431182332 +0000 UTC m=+319.570342922"
Oct 31 07:34:16.405081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:16.404390    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cdbcb4849-t7xbb" event={"ID":"3992d138-a7d9-42f6-b3b7-6dc9e2cdb478","Type":"ContainerStarted","Data":"a1660bf6f3f23502b58320c078fe195f59738cfda1873bf062fda1fb31f1b9b8"}
Oct 31 07:34:16.519490 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:34:16.519441    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:34:17.417362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:17.417316    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j" event={"ID":"ce64b388-ff7c-4cc1-8bea-74e05c0cfd97","Type":"ContainerStarted","Data":"47613cad96524e40af5f48acc760d9d2e046c7ca15c3ad1c296adee508f43d60"}
Oct 31 07:34:17.417786 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:17.417416    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:17.456923 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:17.456867    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j" podStartSLOduration=2.111935253 podStartE2EDuration="4.456848325s" totalImagesPullingTime="2.344913072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:34:17.455635628 +0000 UTC m=+321.594796217" watchObservedRunningTime="2026-10-31 07:34:17.456848325 +0000 UTC m=+321.596008921"
Oct 31 07:34:18.423449 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:18.423404    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s" event={"ID":"60d3af07-65ef-4ee0-b640-f0e8c751440a","Type":"ContainerStarted","Data":"629e3cc387dc65d5b04606c6f8350463ce461654ea4a0631667aa2874fa70602"}
Oct 31 07:34:18.423936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:18.423590    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:18.462822 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:18.462762    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s" podStartSLOduration=1.688948104 podStartE2EDuration="4.462746009s" totalImagesPullingTime="2.773797905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:34:18.461759833 +0000 UTC m=+322.600920421" watchObservedRunningTime="2026-10-31 07:34:18.462746009 +0000 UTC m=+322.601906596"
Oct 31 07:34:28.424530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:28.424498    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"
Oct 31 07:34:29.433386 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:34:29.433355    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5654d67679-dw56s"
Oct 31 07:34:46.510215 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:34:46.510174    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:35:16.510355 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:35:16.510260    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:35:46.510196 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:35:46.510155    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:36:13.169420 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:36:13.169380    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:36:16.509406 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:36:16.509368    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:36:46.510499 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:36:46.510419    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:37:16.509942 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:37:16.509903    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:37:17.659555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:17.659522    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:37:18.895513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.895477    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"]
Oct 31 07:37:18.899232 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.899208    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:18.929510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.929478    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 07:37:18.929510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.929482    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 07:37:18.929510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.929482    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mghfb\""
Oct 31 07:37:18.929510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.929488    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 07:37:18.930940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.930911    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"]
Oct 31 07:37:18.989710 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.989669    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:18.989710 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.989704    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:18.989937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.989730    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:18.989937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.989835    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n85nx\" (UniqueName: \"kubernetes.io/projected/bed58086-ad9a-45b7-a859-65f197560321-kube-api-access-n85nx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:18.989937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.989874    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed58086-ad9a-45b7-a859-65f197560321-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:18.989937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:18.989897    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.090994 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.090964    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n85nx\" (UniqueName: \"kubernetes.io/projected/bed58086-ad9a-45b7-a859-65f197560321-kube-api-access-n85nx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.091186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.091014    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed58086-ad9a-45b7-a859-65f197560321-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.091186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.091040    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.091186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.091071    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.091186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.091086    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.091186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.091112    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.091496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.091476    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.091554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.091536    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.091595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.091552    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.093351 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.093330    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bed58086-ad9a-45b7-a859-65f197560321-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.093501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.093482    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bed58086-ad9a-45b7-a859-65f197560321-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.111092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.111062    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n85nx\" (UniqueName: \"kubernetes.io/projected/bed58086-ad9a-45b7-a859-65f197560321-kube-api-access-n85nx\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b6pmz\" (UID: \"bed58086-ad9a-45b7-a859-65f197560321\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.214987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.214899    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:19.511236 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.511161    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"]
Oct 31 07:37:19.514680 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:37:19.514649    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed58086_ad9a_45b7_a859_65f197560321.slice/crio-e0ceec47e357db797f22ec394ae78cb952c9a9cc1145f214ae07348547c21bec WatchSource:0}: Error finding container e0ceec47e357db797f22ec394ae78cb952c9a9cc1145f214ae07348547c21bec: Status 404 returned error can't find the container with id e0ceec47e357db797f22ec394ae78cb952c9a9cc1145f214ae07348547c21bec
Oct 31 07:37:19.515969 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.515950    5829 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:37:19.685220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:19.685185    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:37:20.266956 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:20.266910    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz" event={"ID":"bed58086-ad9a-45b7-a859-65f197560321","Type":"ContainerStarted","Data":"e0ceec47e357db797f22ec394ae78cb952c9a9cc1145f214ae07348547c21bec"}
Oct 31 07:37:21.273919 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:21.273881    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz" event={"ID":"bed58086-ad9a-45b7-a859-65f197560321","Type":"ContainerStarted","Data":"6a06bb89809bf0343a0de6b1dff050c5aaadcc3d7ce688d03d62f7875edf5d07"}
Oct 31 07:37:21.314219 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:21.314169    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz" podStartSLOduration=2.3133704440000002 podStartE2EDuration="3.314155601s" totalImagesPullingTime="1.000785157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:21.312948281 +0000 UTC m=+505.452108868" watchObservedRunningTime="2026-10-31 07:37:21.314155601 +0000 UTC m=+505.453316188"
Oct 31 07:37:22.216065 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:22.216018    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:23.881186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:23.881149    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:37:24.529368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.529328    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jm2hw"]
Oct 31 07:37:24.533139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.533117    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.565892 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.565860    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 07:37:24.567179 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.567157    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jm2hw"]
Oct 31 07:37:24.645249 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.645214    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.645249 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.645257    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.645533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.645320    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-home\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.645533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.645364    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwnq\" (UniqueName: \"kubernetes.io/projected/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-kube-api-access-ndwnq\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.645533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.645469    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.645741 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.645628    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.747038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.746994    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-home\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.747038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.747040    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwnq\" (UniqueName: \"kubernetes.io/projected/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-kube-api-access-ndwnq\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.747305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.747069    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.747305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.747160    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.747305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.747197    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.747305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.747221    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.747514 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.747460    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-home\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.747514 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.747493    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.747623 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.747609    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.749345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.749324    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.749621 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.749604    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.768078 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.768048    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwnq\" (UniqueName: \"kubernetes.io/projected/cdcc4524-8f99-4e74-8369-90b0b7ca86d5-kube-api-access-ndwnq\") pod \"echo-simulator-kserve-64b6f8f96c-jm2hw\" (UID: \"cdcc4524-8f99-4e74-8369-90b0b7ca86d5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:24.845873 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:24.845787    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:25.142225 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:25.142191    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-jm2hw"]
Oct 31 07:37:25.145581 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:37:25.145555    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcc4524_8f99_4e74_8369_90b0b7ca86d5.slice/crio-7ded047b72ca76919df4d5522a7128881859bfe7d90820f75481f02935139195 WatchSource:0}: Error finding container 7ded047b72ca76919df4d5522a7128881859bfe7d90820f75481f02935139195: Status 404 returned error can't find the container with id 7ded047b72ca76919df4d5522a7128881859bfe7d90820f75481f02935139195
Oct 31 07:37:25.297192 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:25.297156    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw" event={"ID":"cdcc4524-8f99-4e74-8369-90b0b7ca86d5","Type":"ContainerStarted","Data":"7ded047b72ca76919df4d5522a7128881859bfe7d90820f75481f02935139195"}
Oct 31 07:37:27.308905 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:27.308861    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw" event={"ID":"cdcc4524-8f99-4e74-8369-90b0b7ca86d5","Type":"ContainerStarted","Data":"bba3a76cb1aec0abe0bd42f0ed2c178e3a4d4cb63645110765292b2b9c783932"}
Oct 31 07:37:27.349798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:27.349733    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw" podStartSLOduration=1.888299028 podStartE2EDuration="3.349717525s" totalImagesPullingTime="1.461418497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:27.348091937 +0000 UTC m=+511.487252528" watchObservedRunningTime="2026-10-31 07:37:27.349717525 +0000 UTC m=+511.488878110"
Oct 31 07:37:27.846657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:27.846613    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:27.874473 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:27.874440    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:37:29.216049 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.216016    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:29.228449 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.228421    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:29.330258 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.330224    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b6pmz"
Oct 31 07:37:29.800160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.800128    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"]
Oct 31 07:37:29.803810 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.803789    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.822303 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.822245    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 07:37:29.824254 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.824231    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"]
Oct 31 07:37:29.897772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.897730    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.897772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.897774    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.897981 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.897800    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.897981 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.897914    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.898050 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.897999    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfcdq\" (UniqueName: \"kubernetes.io/projected/9271b52d-799d-4bd9-bf35-3c0b466b9346-kube-api-access-bfcdq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.898050 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.898032    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9271b52d-799d-4bd9-bf35-3c0b466b9346-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.999490 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.999456    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcdq\" (UniqueName: \"kubernetes.io/projected/9271b52d-799d-4bd9-bf35-3c0b466b9346-kube-api-access-bfcdq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.999682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.999500    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9271b52d-799d-4bd9-bf35-3c0b466b9346-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.999682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.999538    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.999682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.999558    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.999682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.999597    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:29.999682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.999618    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:30.000023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:29.999996    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:30.000074 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:30.000007    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:30.000136 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:30.000116    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:30.001971 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:30.001947    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9271b52d-799d-4bd9-bf35-3c0b466b9346-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:30.002132 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:30.002116    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9271b52d-799d-4bd9-bf35-3c0b466b9346-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:30.018976 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:30.018946    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcdq\" (UniqueName: \"kubernetes.io/projected/9271b52d-799d-4bd9-bf35-3c0b466b9346-kube-api-access-bfcdq\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6\" (UID: \"9271b52d-799d-4bd9-bf35-3c0b466b9346\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:30.118943 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:30.118852    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:30.416723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:30.416690    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"]
Oct 31 07:37:30.420471 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:37:30.420443    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9271b52d_799d_4bd9_bf35_3c0b466b9346.slice/crio-bfddabb7e590d19063075a533854d5f8610e8bfb216a96f44ea04695d1671522 WatchSource:0}: Error finding container bfddabb7e590d19063075a533854d5f8610e8bfb216a96f44ea04695d1671522: Status 404 returned error can't find the container with id bfddabb7e590d19063075a533854d5f8610e8bfb216a96f44ea04695d1671522
Oct 31 07:37:31.074013 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.073977    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:37:31.330388 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.330307    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6" event={"ID":"9271b52d-799d-4bd9-bf35-3c0b466b9346","Type":"ContainerStarted","Data":"6c915542fad64c6524dcb1a7790d80e83d4c20fd896b05985a89b30baa3bdf6f"}
Oct 31 07:37:31.330388 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.330357    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6" event={"ID":"9271b52d-799d-4bd9-bf35-3c0b466b9346","Type":"ContainerStarted","Data":"bfddabb7e590d19063075a533854d5f8610e8bfb216a96f44ea04695d1671522"}
Oct 31 07:37:31.370467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.370408    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6" podStartSLOduration=2.202980884 podStartE2EDuration="2.37039169s" totalImagesPullingTime="167.410806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:31.369336448 +0000 UTC m=+515.508497036" watchObservedRunningTime="2026-10-31 07:37:31.37039169 +0000 UTC m=+515.509552278"
Oct 31 07:37:31.498875 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.498837    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"]
Oct 31 07:37:31.502591 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.502567    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.520533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.520498    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 07:37:31.520727 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.520711    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"]
Oct 31 07:37:31.615580 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.615499    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.615734 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.615604    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.615734 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.615657    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.615734 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.615729    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.615838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.615814    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.615874 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.615841    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkk6j\" (UniqueName: \"kubernetes.io/projected/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-kube-api-access-qkk6j\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.717434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.717398    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.717628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.717460    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.717628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.717489    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.717628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.717512    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkk6j\" (UniqueName: \"kubernetes.io/projected/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-kube-api-access-qkk6j\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.717628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.717603    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.717855 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.717631    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.717910 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.717880    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.717958 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.717935    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.718003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.717980    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.719859 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.719831    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.720036 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.720017    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.737327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.737302    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkk6j\" (UniqueName: \"kubernetes.io/projected/0b5c0eeb-8829-4fa6-8ce0-617cbf88b831-kube-api-access-qkk6j\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4\" (UID: \"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:31.816292 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:31.816251    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:32.112367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:32.112337    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"]
Oct 31 07:37:32.116143 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:37:32.116119    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b5c0eeb_8829_4fa6_8ce0_617cbf88b831.slice/crio-7c793169ff9078ce9e4ae5c15e8eb79118886465d9206d9e968979fa08e3a8e2 WatchSource:0}: Error finding container 7c793169ff9078ce9e4ae5c15e8eb79118886465d9206d9e968979fa08e3a8e2: Status 404 returned error can't find the container with id 7c793169ff9078ce9e4ae5c15e8eb79118886465d9206d9e968979fa08e3a8e2
Oct 31 07:37:32.119170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:32.119148    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:32.336912 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:32.336881    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" event={"ID":"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831","Type":"ContainerStarted","Data":"7c793169ff9078ce9e4ae5c15e8eb79118886465d9206d9e968979fa08e3a8e2"}
Oct 31 07:37:33.343220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:33.343185    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" event={"ID":"0b5c0eeb-8829-4fa6-8ce0-617cbf88b831","Type":"ContainerStarted","Data":"0a0dc17eeff5615d971203dcc29728a9381ecadba86774d63fd19becc892db07"}
Oct 31 07:37:33.388687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:33.388631    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podStartSLOduration=2.217324711 podStartE2EDuration="2.388614973s" totalImagesPullingTime="171.290262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:33.387274364 +0000 UTC m=+517.526434952" watchObservedRunningTime="2026-10-31 07:37:33.388614973 +0000 UTC m=+517.527775561"
Oct 31 07:37:33.816931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:33.816895    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:34.845992 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:34.845948    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:34.858526 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:34.858495    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:35.363357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:35.363322    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-jm2hw"
Oct 31 07:37:35.599419 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:35.599387    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:37:40.119330 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:40.119261    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:40.131847 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:40.131819    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:40.387535 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:40.387514    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6"
Oct 31 07:37:41.048837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.048796    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"]
Oct 31 07:37:41.052873 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.052847    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.067932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.067907    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 07:37:41.080987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.080956    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-m48c6\""
Oct 31 07:37:41.083311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.082766    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"]
Oct 31 07:37:41.106119 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.106077    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.106315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.106148    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.106315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.106263    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.106449 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.106331    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2991db1f-d2e3-428f-8746-ec01ace2e05b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.106504 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.106472    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.106566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.106539    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.106600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.106578    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.106641 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.106608    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8p7\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-kube-api-access-5p8p7\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.106641 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.106634    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.207749 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.207716    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.207758    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2991db1f-d2e3-428f-8746-ec01ace2e05b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.207800    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.207821    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.207840    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.207864    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8p7\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-kube-api-access-5p8p7\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.207893    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.207931    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.208185    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.208233    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.208257    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.208276    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.208362    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.208755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.208686    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2991db1f-d2e3-428f-8746-ec01ace2e05b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.210323 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.210305    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.210477 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.210461    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.227333 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.227274    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.227460 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.227415    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8p7\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-kube-api-access-5p8p7\") pod \"maas-default-gateway-openshift-default-645cf464d7-vt92f\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.366343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.366225    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:41.679337 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.679302    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"]
Oct 31 07:37:41.686609 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:37:41.686582    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2991db1f_d2e3_428f_8746_ec01ace2e05b.slice/crio-a9d8ca6c5ea6c65cbb0dddb2449a8b8b3056226e2712c15b58dc53ab92262893 WatchSource:0}: Error finding container a9d8ca6c5ea6c65cbb0dddb2449a8b8b3056226e2712c15b58dc53ab92262893: Status 404 returned error can't find the container with id a9d8ca6c5ea6c65cbb0dddb2449a8b8b3056226e2712c15b58dc53ab92262893
Oct 31 07:37:41.817136 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.817102    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:41.829670 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:41.829646    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:42.388134 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:42.388096    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" event={"ID":"2991db1f-d2e3-428f-8746-ec01ace2e05b","Type":"ContainerStarted","Data":"a9d8ca6c5ea6c65cbb0dddb2449a8b8b3056226e2712c15b58dc53ab92262893"}
Oct 31 07:37:42.403086 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:42.403055    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:37:44.242920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:44.242886    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:37:44.243212 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:44.242960    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:37:44.243212 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:44.242991    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:37:44.403423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:44.403387    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" event={"ID":"2991db1f-d2e3-428f-8746-ec01ace2e05b","Type":"ContainerStarted","Data":"dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c"}
Oct 31 07:37:44.471046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:44.470988    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podStartSLOduration=0.917498235 podStartE2EDuration="3.470966722s" totalImagesPullingTime="2.553468487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:44.457051573 +0000 UTC m=+528.596212162" watchObservedRunningTime="2026-10-31 07:37:44.470966722 +0000 UTC m=+528.610127309"
Oct 31 07:37:45.367372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:45.367336    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:45.368959 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:45.368932    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-vt92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 07:37:45.369084 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:45.368985    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 07:37:46.366862 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:46.366827    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-vt92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 07:37:46.367042 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:46.366883    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 07:37:46.537023 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:37:46.536927    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:37:47.366923 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:47.366887    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-vt92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 07:37:47.367096 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:47.366959    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 07:37:48.367269 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:48.367237    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-vt92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 07:37:48.367706 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:48.367312    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 07:37:49.366615 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:49.366577    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-vt92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 07:37:49.366857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:49.366635    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 07:37:50.366686 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:50.366651    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-vt92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 07:37:50.367079 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:50.366704    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 07:37:51.367492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:51.367456    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-vt92f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 07:37:51.367883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:51.367514    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 07:37:52.508119 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:52.508084    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:52.508549 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:52.508395    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:52.509059 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:52.509034    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 07:37:53.728880 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.728841    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"]
Oct 31 07:37:53.735465 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.735431    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:37:53.752442 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.752411    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"]
Oct 31 07:37:53.762056 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.762020    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 07:37:53.762056 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.762020    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 07:37:53.763560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.763525    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 07:37:53.763560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.763538    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 07:37:53.763718 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.763564    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-t4ppz\""
Oct 31 07:37:53.832950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.832914    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbdkb\" (UniqueName: \"kubernetes.io/projected/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-kube-api-access-gbdkb\") pod \"ai-gateway-controller-7d6c696b49-ss7vq\" (UID: \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:37:53.833121 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.832975    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-ss7vq\" (UID: \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:37:53.934575 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.934536    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbdkb\" (UniqueName: \"kubernetes.io/projected/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-kube-api-access-gbdkb\") pod \"ai-gateway-controller-7d6c696b49-ss7vq\" (UID: \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:37:53.934756 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.934583    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-ss7vq\" (UID: \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:37:53.937001 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.936971    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-metrics-certs\") pod \"ai-gateway-controller-7d6c696b49-ss7vq\" (UID: \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:37:53.954032 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:53.954007    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbdkb\" (UniqueName: \"kubernetes.io/projected/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-kube-api-access-gbdkb\") pod \"ai-gateway-controller-7d6c696b49-ss7vq\" (UID: \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\") " pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:37:54.053989 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:54.053885    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:37:54.354735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:54.354709    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"]
Oct 31 07:37:54.357937 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:37:54.357904    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb633a5b0_bc60_4fa6_931b_e3fb9af9cf52.slice/crio-0b982a94a850531831848593dab2a2d4542959be60323baceb9697c0821006cd WatchSource:0}: Error finding container 0b982a94a850531831848593dab2a2d4542959be60323baceb9697c0821006cd: Status 404 returned error can't find the container with id 0b982a94a850531831848593dab2a2d4542959be60323baceb9697c0821006cd
Oct 31 07:37:54.478476 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:54.478437    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq" event={"ID":"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52","Type":"ContainerStarted","Data":"0b982a94a850531831848593dab2a2d4542959be60323baceb9697c0821006cd"}
Oct 31 07:37:56.048324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.048276    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"]
Oct 31 07:37:56.068213 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.068170    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.077199 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.077168    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"]
Oct 31 07:37:56.159796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.159761    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.159978 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.159816    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305fd353-98c5-4686-b2b2-b7ee1a752d87-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.159978 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.159933    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.159978 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.159968    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.160148 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.160000    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.160148 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.160087    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.160250 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.160147    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.160250 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.160223    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.160386 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.160255    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qvv\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-kube-api-access-97qvv\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.260968 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.260938    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.260985    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261015    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261046    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261083    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261111    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261134    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qvv\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-kube-api-access-97qvv\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261479 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261230    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261479 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261276    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305fd353-98c5-4686-b2b2-b7ee1a752d87-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261479 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261359    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261479 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261444    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261684    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.261847 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.261730    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.262026 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.262006    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305fd353-98c5-4686-b2b2-b7ee1a752d87-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.263761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.263737    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.263918 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.263899    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.280482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.280453    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.280614 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.280492    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qvv\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-kube-api-access-97qvv\") pod \"maas-default-gateway-openshift-default-645cf464d7-hkpt2\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.390630 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.390597    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:56.714511 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.714436    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"]
Oct 31 07:37:56.717520 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:37:56.717488    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305fd353_98c5_4686_b2b2_b7ee1a752d87.slice/crio-9cb192a53408ccc274d45dffdd96768ece7ffa5267d415a998421430ca038447 WatchSource:0}: Error finding container 9cb192a53408ccc274d45dffdd96768ece7ffa5267d415a998421430ca038447: Status 404 returned error can't find the container with id 9cb192a53408ccc274d45dffdd96768ece7ffa5267d415a998421430ca038447
Oct 31 07:37:56.719897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.719864    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:37:56.719970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.719941    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:37:56.720004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.719971    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:37:56.743028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:56.743007    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 07:37:57.503137 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:57.503102    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq" event={"ID":"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52","Type":"ContainerStarted","Data":"5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a"}
Oct 31 07:37:57.503656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:57.503185    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:37:57.522306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:57.514637    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" event={"ID":"305fd353-98c5-4686-b2b2-b7ee1a752d87","Type":"ContainerStarted","Data":"fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6"}
Oct 31 07:37:57.522306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:57.514719    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" event={"ID":"305fd353-98c5-4686-b2b2-b7ee1a752d87","Type":"ContainerStarted","Data":"9cb192a53408ccc274d45dffdd96768ece7ffa5267d415a998421430ca038447"}
Oct 31 07:37:57.552238 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:57.552182    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq" podStartSLOduration=2.183951461 podStartE2EDuration="4.552169492s" totalImagesPullingTime="2.368218031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:57.551178383 +0000 UTC m=+541.690338971" watchObservedRunningTime="2026-10-31 07:37:57.552169492 +0000 UTC m=+541.691330080"
Oct 31 07:37:57.594534 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:57.594475    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podStartSLOduration=1.594459465 podStartE2EDuration="1.594459465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:37:57.592545829 +0000 UTC m=+541.731706414" watchObservedRunningTime="2026-10-31 07:37:57.594459465 +0000 UTC m=+541.733620052"
Oct 31 07:37:58.147981 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.147948    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"]
Oct 31 07:37:58.151804 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.151782    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:37:58.171311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.169928    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 07:37:58.171311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.170084    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xnqjm\""
Oct 31 07:37:58.172248 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.172224    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"]
Oct 31 07:37:58.237272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.237231    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-sp8xc"]
Oct 31 07:37:58.241043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.241019    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:37:58.259949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.259915    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-sp8xc"]
Oct 31 07:37:58.282596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.282549    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct584\" (UniqueName: \"kubernetes.io/projected/2b71dae3-c259-4a13-af86-ab549638b3d8-kube-api-access-ct584\") pod \"payload-pre-processing-7b98bb678c-rw25z\" (UID: \"2b71dae3-c259-4a13-af86-ab549638b3d8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:37:58.282790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.282621    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b71dae3-c259-4a13-af86-ab549638b3d8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rw25z\" (UID: \"2b71dae3-c259-4a13-af86-ab549638b3d8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:37:58.384019 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.383976    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrww\" (UniqueName: \"kubernetes.io/projected/02eee9ac-9709-4b6b-a26c-3fd5ea5e1658-kube-api-access-hbrww\") pod \"payload-processing-7bb9489676-sp8xc\" (UID: \"02eee9ac-9709-4b6b-a26c-3fd5ea5e1658\") " pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:37:58.384197 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.384131    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct584\" (UniqueName: \"kubernetes.io/projected/2b71dae3-c259-4a13-af86-ab549638b3d8-kube-api-access-ct584\") pod \"payload-pre-processing-7b98bb678c-rw25z\" (UID: \"2b71dae3-c259-4a13-af86-ab549638b3d8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:37:58.384197 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.384173    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b71dae3-c259-4a13-af86-ab549638b3d8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rw25z\" (UID: \"2b71dae3-c259-4a13-af86-ab549638b3d8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:37:58.384372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.384223    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02eee9ac-9709-4b6b-a26c-3fd5ea5e1658-plugins-config-volume\") pod \"payload-processing-7bb9489676-sp8xc\" (UID: \"02eee9ac-9709-4b6b-a26c-3fd5ea5e1658\") " pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:37:58.384918 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.384898    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b71dae3-c259-4a13-af86-ab549638b3d8-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-rw25z\" (UID: \"2b71dae3-c259-4a13-af86-ab549638b3d8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:37:58.390927 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.390904    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:37:58.392550 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.392531    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 31 07:37:58.392643 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.392574    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 31 07:37:58.433351 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.432522    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct584\" (UniqueName: \"kubernetes.io/projected/2b71dae3-c259-4a13-af86-ab549638b3d8-kube-api-access-ct584\") pod \"payload-pre-processing-7b98bb678c-rw25z\" (UID: \"2b71dae3-c259-4a13-af86-ab549638b3d8\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:37:58.465395 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.465361    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:37:58.484951 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.484923    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrww\" (UniqueName: \"kubernetes.io/projected/02eee9ac-9709-4b6b-a26c-3fd5ea5e1658-kube-api-access-hbrww\") pod \"payload-processing-7bb9489676-sp8xc\" (UID: \"02eee9ac-9709-4b6b-a26c-3fd5ea5e1658\") " pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:37:58.485083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.485017    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02eee9ac-9709-4b6b-a26c-3fd5ea5e1658-plugins-config-volume\") pod \"payload-processing-7bb9489676-sp8xc\" (UID: \"02eee9ac-9709-4b6b-a26c-3fd5ea5e1658\") " pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:37:58.485965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.485937    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02eee9ac-9709-4b6b-a26c-3fd5ea5e1658-plugins-config-volume\") pod \"payload-processing-7bb9489676-sp8xc\" (UID: \"02eee9ac-9709-4b6b-a26c-3fd5ea5e1658\") " pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:37:58.534563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.525644    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrww\" (UniqueName: \"kubernetes.io/projected/02eee9ac-9709-4b6b-a26c-3fd5ea5e1658-kube-api-access-hbrww\") pod \"payload-processing-7bb9489676-sp8xc\" (UID: \"02eee9ac-9709-4b6b-a26c-3fd5ea5e1658\") " pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:37:58.554551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.554523    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:37:58.798848 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.798821    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"]
Oct 31 07:37:58.801561 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:37:58.801528    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b71dae3_c259_4a13_af86_ab549638b3d8.slice/crio-34d53f5329b12d67d92b40134acc1b4ae573c06ba72832dca5dbee2e968804e1 WatchSource:0}: Error finding container 34d53f5329b12d67d92b40134acc1b4ae573c06ba72832dca5dbee2e968804e1: Status 404 returned error can't find the container with id 34d53f5329b12d67d92b40134acc1b4ae573c06ba72832dca5dbee2e968804e1
Oct 31 07:37:58.887454 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:58.887424    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-7bb9489676-sp8xc"]
Oct 31 07:37:58.891478 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:37:58.891443    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eee9ac_9709_4b6b_a26c_3fd5ea5e1658.slice/crio-d03afef94fc16402fd95b05e99539a09c64e564d0a8fc5b3e35b613b8c7f3a2a WatchSource:0}: Error finding container d03afef94fc16402fd95b05e99539a09c64e564d0a8fc5b3e35b613b8c7f3a2a: Status 404 returned error can't find the container with id d03afef94fc16402fd95b05e99539a09c64e564d0a8fc5b3e35b613b8c7f3a2a
Oct 31 07:37:59.391409 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:59.391370    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 31 07:37:59.391586 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:59.391437    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 31 07:37:59.548890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:59.548747    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z" event={"ID":"2b71dae3-c259-4a13-af86-ab549638b3d8","Type":"ContainerStarted","Data":"34d53f5329b12d67d92b40134acc1b4ae573c06ba72832dca5dbee2e968804e1"}
Oct 31 07:37:59.551713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:37:59.551650    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-sp8xc" event={"ID":"02eee9ac-9709-4b6b-a26c-3fd5ea5e1658","Type":"ContainerStarted","Data":"d03afef94fc16402fd95b05e99539a09c64e564d0a8fc5b3e35b613b8c7f3a2a"}
Oct 31 07:38:00.392570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:00.392512    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 31 07:38:00.393041 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:00.392933    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 31 07:38:01.391570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:01.391533    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 31 07:38:01.392065 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:01.391597    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 31 07:38:02.391864 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:02.391815    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 31 07:38:02.392354 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:02.391878    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 31 07:38:03.391002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:03.390970    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 31 07:38:03.391189 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:03.391022    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 31 07:38:03.580037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:03.579998    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-7bb9489676-sp8xc" event={"ID":"02eee9ac-9709-4b6b-a26c-3fd5ea5e1658","Type":"ContainerStarted","Data":"f038078c60da23983d8c83c5891355285b5918b865428397a988bed8e769162d"}
Oct 31 07:38:03.580569 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:03.580183    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:38:03.581736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:03.581709    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z" event={"ID":"2b71dae3-c259-4a13-af86-ab549638b3d8","Type":"ContainerStarted","Data":"f002bc461b162ee6b9ad7121a714cdd08041d13d86b4c9bf4ebbd717dd804aaa"}
Oct 31 07:38:03.581860 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:03.581837    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:38:03.620716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:03.620653    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-7bb9489676-sp8xc" podStartSLOduration=2.043255973 podStartE2EDuration="5.620632831s" totalImagesPullingTime="3.577376858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:03.618907523 +0000 UTC m=+547.758068114" watchObservedRunningTime="2026-10-31 07:38:03.620632831 +0000 UTC m=+547.759793422"
Oct 31 07:38:03.658930 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:03.658808    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z" podStartSLOduration=1.9210413210000001 podStartE2EDuration="5.658790744s" totalImagesPullingTime="3.737749423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:03.656743268 +0000 UTC m=+547.795903857" watchObservedRunningTime="2026-10-31 07:38:03.658790744 +0000 UTC m=+547.797951332"
Oct 31 07:38:04.391139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:04.391102    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Oct 31 07:38:04.391325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:04.391163    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Oct 31 07:38:06.432508 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:06.432462    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:38:06.433045 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:06.432540    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:38:07.297630 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:07.297589    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:38:07.298036 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:07.298009    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:38:07.299252 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:07.299231    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 07:38:08.538970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.538884    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:38:08.809077 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.808998    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"]
Oct 31 07:38:08.812916 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.812898    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:08.825451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.825425    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jp2\" (UniqueName: \"kubernetes.io/projected/56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5-kube-api-access-58jp2\") pod \"ai-gateway-controller-66fd44d76c-d9rzk\" (UID: \"56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5\") " pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:08.825560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.825466    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5-metrics-certs\") pod \"ai-gateway-controller-66fd44d76c-d9rzk\" (UID: \"56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5\") " pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:08.830029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.830006    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"]
Oct 31 07:38:08.926416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.926384    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jp2\" (UniqueName: \"kubernetes.io/projected/56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5-kube-api-access-58jp2\") pod \"ai-gateway-controller-66fd44d76c-d9rzk\" (UID: \"56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5\") " pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:08.926606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.926432    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5-metrics-certs\") pod \"ai-gateway-controller-66fd44d76c-d9rzk\" (UID: \"56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5\") " pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:08.929011 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.928981    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5-metrics-certs\") pod \"ai-gateway-controller-66fd44d76c-d9rzk\" (UID: \"56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5\") " pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:08.946133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:08.946098    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jp2\" (UniqueName: \"kubernetes.io/projected/56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5-kube-api-access-58jp2\") pod \"ai-gateway-controller-66fd44d76c-d9rzk\" (UID: \"56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5\") " pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:09.125320 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:09.125223    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:09.423339 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:09.423316    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"]
Oct 31 07:38:09.425997 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:09.425970    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f3ce14_6bd4_4aea_8d3b_6bc012bdc7e5.slice/crio-99f2649fedcff85cbfc5126e36de7ef060538f4b71089bc720c454b37ba07846 WatchSource:0}: Error finding container 99f2649fedcff85cbfc5126e36de7ef060538f4b71089bc720c454b37ba07846: Status 404 returned error can't find the container with id 99f2649fedcff85cbfc5126e36de7ef060538f4b71089bc720c454b37ba07846
Oct 31 07:38:09.615308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:09.615259    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk" event={"ID":"56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5","Type":"ContainerStarted","Data":"99f2649fedcff85cbfc5126e36de7ef060538f4b71089bc720c454b37ba07846"}
Oct 31 07:38:10.622294 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:10.622250    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk" event={"ID":"56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5","Type":"ContainerStarted","Data":"b8c62a674c8c2b6dce2eb139f39358b3ba514aeb4f9c405debfc61c7a13f7a35"}
Oct 31 07:38:10.622723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:10.622343    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:10.662153 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:10.662100    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk" podStartSLOduration=2.3776879380000002 podStartE2EDuration="2.662080535s" totalImagesPullingTime="284.392597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:10.661152015 +0000 UTC m=+554.800312607" watchObservedRunningTime="2026-10-31 07:38:10.662080535 +0000 UTC m=+554.801241122"
Oct 31 07:38:14.590535 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:14.590496    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-rw25z"
Oct 31 07:38:14.590988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:14.590570    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-7bb9489676-sp8xc"
Oct 31 07:38:16.509796 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:38:16.509759    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:38:21.629655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:21.629619    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-66fd44d76c-d9rzk"
Oct 31 07:38:21.689752 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:21.689714    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"]
Oct 31 07:38:21.690066 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:21.690035    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq" podUID="b633a5b0-bc60-4fa6-931b-e3fb9af9cf52" containerName="manager" containerID="cri-o://5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a" gracePeriod=30
Oct 31 07:38:21.941573 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:21.941542    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:38:22.051911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.051872    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-metrics-certs\") pod \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\" (UID: \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\") "
Oct 31 07:38:22.052081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.052053    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-kube-api-access-gbdkb\" (UniqueName: \"kubernetes.io/projected/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-kube-api-access-gbdkb\") pod \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\" (UID: \"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52\") "
Oct 31 07:38:22.054111 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.054078    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-metrics-certs" pod "b633a5b0-bc60-4fa6-931b-e3fb9af9cf52" (UID: "b633a5b0-bc60-4fa6-931b-e3fb9af9cf52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:38:22.054238 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.054155    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-kube-api-access-gbdkb" pod "b633a5b0-bc60-4fa6-931b-e3fb9af9cf52" (UID: "b633a5b0-bc60-4fa6-931b-e3fb9af9cf52"). InnerVolumeSpecName "kube-api-access-gbdkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:38:22.152906 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.152859    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbdkb\" (UniqueName: \"kubernetes.io/projected/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-kube-api-access-gbdkb\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:38:22.152906 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.152897    5829 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52-metrics-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:38:22.688368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.688325    5829 generic.go:356] "Generic (PLEG): container finished" podID="b633a5b0-bc60-4fa6-931b-e3fb9af9cf52" containerID="5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a" exitCode=0
Oct 31 07:38:22.688783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.688398    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"
Oct 31 07:38:22.688783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.688409    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq" event={"ID":"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52","Type":"ContainerDied","Data":"5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a"}
Oct 31 07:38:22.688783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.688494    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d6c696b49-ss7vq" event={"ID":"b633a5b0-bc60-4fa6-931b-e3fb9af9cf52","Type":"ContainerDied","Data":"0b982a94a850531831848593dab2a2d4542959be60323baceb9697c0821006cd"}
Oct 31 07:38:22.688783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.688520    5829 scope.go:122] "RemoveContainer" containerID="5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a"
Oct 31 07:38:22.698046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.697979    5829 scope.go:122] "RemoveContainer" containerID="5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a"
Oct 31 07:38:22.698365 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:38:22.698344    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a\": container with ID starting with 5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a not found: ID does not exist" containerID="5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a"
Oct 31 07:38:22.698407 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.698375    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a"} err="failed to get container status \"5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a\": rpc error: code = NotFound desc = could not find container \"5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a\": container with ID starting with 5be01d86e78bf665e32126a330705a23497696feb153954840ab9fb10e7ec70a not found: ID does not exist"
Oct 31 07:38:22.740111 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.740068    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"]
Oct 31 07:38:22.753468 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:22.753442    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d6c696b49-ss7vq"]
Oct 31 07:38:23.972791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:23.972754    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b633a5b0-bc60-4fa6-931b-e3fb9af9cf52" path="/var/lib/kubelet/pods/b633a5b0-bc60-4fa6-931b-e3fb9af9cf52/volumes"
Oct 31 07:38:46.509805 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:38:46.509769    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:38:47.776581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.776540    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"]
Oct 31 07:38:47.777080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.776997    5829 cpu_manager.go:414] "containerMap: removing container" podUID="b633a5b0-bc60-4fa6-931b-e3fb9af9cf52" containerName="manager"
Oct 31 07:38:47.777080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.777009    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b633a5b0-bc60-4fa6-931b-e3fb9af9cf52" containerName="manager"
Oct 31 07:38:47.777196 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.777097    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="b633a5b0-bc60-4fa6-931b-e3fb9af9cf52" containerName="manager"
Oct 31 07:38:47.780536 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.780513    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.800270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.800243    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-027d2a-openshift-default-dockercfg-7m2tr\""
Oct 31 07:38:47.815049 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.815019    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"]
Oct 31 07:38:47.817257 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.817234    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-token\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.817423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.817271    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-podinfo\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.817423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.817312    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc8nd\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-kube-api-access-xc8nd\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.817530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.817446    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d130a1c7-6b3f-4168-af6a-451147ed223d-istiod-ca-cert\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.817530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.817520    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-certs\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.817632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.817565    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.817632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.817607    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-credential-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.817632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.817628    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-data\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.817786 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.817685    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-envoy\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.918658 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.918608    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-credential-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.918658 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.918664    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-data\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.918999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.918736    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-envoy\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.918999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.918764    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-token\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.918999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.918804    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-podinfo\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.918999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.918842    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc8nd\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-kube-api-access-xc8nd\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.918999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.918900    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d130a1c7-6b3f-4168-af6a-451147ed223d-istiod-ca-cert\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.918999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.918954    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-certs\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.919335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.918998    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.919335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.919102    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-data\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.919488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.919461    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-credential-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.919648 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.919620    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-certs\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.919775 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.919757    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d130a1c7-6b3f-4168-af6a-451147ed223d-istiod-ca-cert\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.920038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.920015    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.921604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.921580    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-podinfo\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.921708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.921606    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-envoy\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.939471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.939439    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc8nd\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-kube-api-access-xc8nd\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:47.940128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:47.940098    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-token\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:48.093655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.093565    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:48.126468 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.126427    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"]
Oct 31 07:38:48.131421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.131385    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.148379 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.148343    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-b1fda4-openshift-default-dockercfg-9xncc\""
Oct 31 07:38:48.151676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.151641    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"]
Oct 31 07:38:48.221346 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.221306    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-data\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.221346 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.221354    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-podinfo\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.221617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.221377    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lzk\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-kube-api-access-26lzk\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.221617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.221473    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.221617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.221519    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-envoy\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.221617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.221542    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef061dbf-d524-4e2b-a234-f14abfd09111-istiod-ca-cert\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.221818 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.221650    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-credential-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.221818 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.221686    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-certs\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.221818 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.221715    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-token\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.323152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.323114    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-envoy\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.323434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.323166    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef061dbf-d524-4e2b-a234-f14abfd09111-istiod-ca-cert\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.323434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.323221    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-credential-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.323434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.323266    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-certs\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.323434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.323331    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-token\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.323434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.323394    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-data\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.323434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.323426    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-podinfo\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.323748 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.323455    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lzk\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-kube-api-access-26lzk\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.323748 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.323492    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.324357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.324001    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-certs\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.324357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.324155    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-data\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.324357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.324234    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.324581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.324521    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-credential-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.324632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.324403    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef061dbf-d524-4e2b-a234-f14abfd09111-istiod-ca-cert\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.327309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.326354    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-envoy\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.334137 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.334113    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-podinfo\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.344905 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.344844    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-token\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.345095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.345077    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lzk\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-kube-api-access-26lzk\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.422999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.422963    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"]
Oct 31 07:38:48.426346 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:48.426320    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd130a1c7_6b3f_4168_af6a_451147ed223d.slice/crio-272488492fcdc251af1386b3ad5e177214bc281a20b8721e04fd295c3e3e29bc WatchSource:0}: Error finding container 272488492fcdc251af1386b3ad5e177214bc281a20b8721e04fd295c3e3e29bc: Status 404 returned error can't find the container with id 272488492fcdc251af1386b3ad5e177214bc281a20b8721e04fd295c3e3e29bc
Oct 31 07:38:48.428695 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.428657    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:38:48.428798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.428731    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:38:48.428798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.428761    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:38:48.448797 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.448774    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:48.772821 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.772673    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"]
Oct 31 07:38:48.775658 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:48.775633    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef061dbf_d524_4e2b_a234_f14abfd09111.slice/crio-7498e2eacda9c5f883bca434fbb239f9d7447996bd98f3357d54cfce138e5bcf WatchSource:0}: Error finding container 7498e2eacda9c5f883bca434fbb239f9d7447996bd98f3357d54cfce138e5bcf: Status 404 returned error can't find the container with id 7498e2eacda9c5f883bca434fbb239f9d7447996bd98f3357d54cfce138e5bcf
Oct 31 07:38:48.778045 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.778013    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:38:48.778348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.778076    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:38:48.778348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.778104    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:38:48.824462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.824429    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" event={"ID":"ef061dbf-d524-4e2b-a234-f14abfd09111","Type":"ContainerStarted","Data":"7498e2eacda9c5f883bca434fbb239f9d7447996bd98f3357d54cfce138e5bcf"}
Oct 31 07:38:48.826219 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.826189    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" event={"ID":"d130a1c7-6b3f-4168-af6a-451147ed223d","Type":"ContainerStarted","Data":"4ab8a0adf6b0a59509566d296e59ce0e87751018e803a6cf57b5723a48ccdd39"}
Oct 31 07:38:48.826319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.826238    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" event={"ID":"d130a1c7-6b3f-4168-af6a-451147ed223d","Type":"ContainerStarted","Data":"272488492fcdc251af1386b3ad5e177214bc281a20b8721e04fd295c3e3e29bc"}
Oct 31 07:38:48.873103 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:48.873044    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" podStartSLOduration=1.8730261910000001 podStartE2EDuration="1.873026191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:48.870276338 +0000 UTC m=+593.009436939" watchObservedRunningTime="2026-10-31 07:38:48.873026191 +0000 UTC m=+593.012186779"
Oct 31 07:38:49.094354 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:49.094244    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:49.099632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:49.099602    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:49.833233 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:49.833196    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" event={"ID":"ef061dbf-d524-4e2b-a234-f14abfd09111","Type":"ContainerStarted","Data":"1b806d8aaf8d2d23f88775832188ac68d5fc3995012195c09a3f010502f1f8d7"}
Oct 31 07:38:49.833676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:49.833487    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:49.834294 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:49.834267    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:38:49.892202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:49.892147    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" podStartSLOduration=1.892133441 podStartE2EDuration="1.892133441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:49.890226311 +0000 UTC m=+594.029386899" watchObservedRunningTime="2026-10-31 07:38:49.892133441 +0000 UTC m=+594.031294028"
Oct 31 07:38:50.449161 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:50.449124    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:50.453937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:50.453910    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:50.838997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:50.838909    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:50.839790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:50.839766    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:38:52.981791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:52.981755    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"]
Oct 31 07:38:52.985249 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:52.985232    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:38:53.004358 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.004327    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"]
Oct 31 07:38:53.015154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.015127    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-027d2a\""
Oct 31 07:38:53.015319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.015259    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-027d2a-dockercfg-8q55c\""
Oct 31 07:38:53.069226 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.069188    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"]
Oct 31 07:38:53.073684 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.073662    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:38:53.074027 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.074002    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e369c6a-79e7-4aed-a1c3-a756e512372c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v\" (UID: \"6e369c6a-79e7-4aed-a1c3-a756e512372c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:38:53.074215 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.074196    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmrmf\" (UniqueName: \"kubernetes.io/projected/6e369c6a-79e7-4aed-a1c3-a756e512372c-kube-api-access-jmrmf\") pod \"payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v\" (UID: \"6e369c6a-79e7-4aed-a1c3-a756e512372c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:38:53.093405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.093377    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"]
Oct 31 07:38:53.175695 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.175657    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53782e9d-d7b0-4a17-b756-f14960c29c2c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2\" (UID: \"53782e9d-d7b0-4a17-b756-f14960c29c2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:38:53.175877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.175705    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8scc\" (UniqueName: \"kubernetes.io/projected/53782e9d-d7b0-4a17-b756-f14960c29c2c-kube-api-access-z8scc\") pod \"payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2\" (UID: \"53782e9d-d7b0-4a17-b756-f14960c29c2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:38:53.175877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.175858    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmrmf\" (UniqueName: \"kubernetes.io/projected/6e369c6a-79e7-4aed-a1c3-a756e512372c-kube-api-access-jmrmf\") pod \"payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v\" (UID: \"6e369c6a-79e7-4aed-a1c3-a756e512372c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:38:53.175972 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.175927    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e369c6a-79e7-4aed-a1c3-a756e512372c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v\" (UID: \"6e369c6a-79e7-4aed-a1c3-a756e512372c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:38:53.176692 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.176669    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e369c6a-79e7-4aed-a1c3-a756e512372c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v\" (UID: \"6e369c6a-79e7-4aed-a1c3-a756e512372c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:38:53.195687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.195657    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmrmf\" (UniqueName: \"kubernetes.io/projected/6e369c6a-79e7-4aed-a1c3-a756e512372c-kube-api-access-jmrmf\") pod \"payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v\" (UID: \"6e369c6a-79e7-4aed-a1c3-a756e512372c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:38:53.276792 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.276703    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53782e9d-d7b0-4a17-b756-f14960c29c2c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2\" (UID: \"53782e9d-d7b0-4a17-b756-f14960c29c2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:38:53.276792 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.276758    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8scc\" (UniqueName: \"kubernetes.io/projected/53782e9d-d7b0-4a17-b756-f14960c29c2c-kube-api-access-z8scc\") pod \"payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2\" (UID: \"53782e9d-d7b0-4a17-b756-f14960c29c2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:38:53.277428 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.277408    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53782e9d-d7b0-4a17-b756-f14960c29c2c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2\" (UID: \"53782e9d-d7b0-4a17-b756-f14960c29c2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:38:53.296714 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.296682    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8scc\" (UniqueName: \"kubernetes.io/projected/53782e9d-d7b0-4a17-b756-f14960c29c2c-kube-api-access-z8scc\") pod \"payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2\" (UID: \"53782e9d-d7b0-4a17-b756-f14960c29c2c\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:38:53.298453 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.298432    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:38:53.386222 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.385975    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:38:53.608737 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.608713    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"]
Oct 31 07:38:53.613415 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:53.613385    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e369c6a_79e7_4aed_a1c3_a756e512372c.slice/crio-1329df141b09de736af671f67f7786633e30f50bd5dc598dd336bc4bc867a9a0 WatchSource:0}: Error finding container 1329df141b09de736af671f67f7786633e30f50bd5dc598dd336bc4bc867a9a0: Status 404 returned error can't find the container with id 1329df141b09de736af671f67f7786633e30f50bd5dc598dd336bc4bc867a9a0
Oct 31 07:38:53.693382 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.693356    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"]
Oct 31 07:38:53.696047 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:53.696019    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53782e9d_d7b0_4a17_b756_f14960c29c2c.slice/crio-9112a856d9e06b69379c7b2d05ad81ddb4579a7cff25af20c8b5a6de5092dbf9 WatchSource:0}: Error finding container 9112a856d9e06b69379c7b2d05ad81ddb4579a7cff25af20c8b5a6de5092dbf9: Status 404 returned error can't find the container with id 9112a856d9e06b69379c7b2d05ad81ddb4579a7cff25af20c8b5a6de5092dbf9
Oct 31 07:38:53.856929 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.856892    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v" event={"ID":"6e369c6a-79e7-4aed-a1c3-a756e512372c","Type":"ContainerStarted","Data":"4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac"}
Oct 31 07:38:53.857123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.856965    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:38:53.857123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.856991    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v" event={"ID":"6e369c6a-79e7-4aed-a1c3-a756e512372c","Type":"ContainerStarted","Data":"1329df141b09de736af671f67f7786633e30f50bd5dc598dd336bc4bc867a9a0"}
Oct 31 07:38:53.859325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.859241    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2" event={"ID":"53782e9d-d7b0-4a17-b756-f14960c29c2c","Type":"ContainerStarted","Data":"7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76"}
Oct 31 07:38:53.859325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.859307    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2" event={"ID":"53782e9d-d7b0-4a17-b756-f14960c29c2c","Type":"ContainerStarted","Data":"9112a856d9e06b69379c7b2d05ad81ddb4579a7cff25af20c8b5a6de5092dbf9"}
Oct 31 07:38:53.859467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.859333    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:38:53.918993 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.918932    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v" podStartSLOduration=1.918916729 podStartE2EDuration="1.918916729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:53.917121053 +0000 UTC m=+598.056281643" watchObservedRunningTime="2026-10-31 07:38:53.918916729 +0000 UTC m=+598.058077316"
Oct 31 07:38:53.961239 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:53.961181    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2" podStartSLOduration=0.961165849 podStartE2EDuration="961.165849ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:53.95917192 +0000 UTC m=+598.098332507" watchObservedRunningTime="2026-10-31 07:38:53.961165849 +0000 UTC m=+598.100326437"
Oct 31 07:38:54.655266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.655230    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"]
Oct 31 07:38:54.659233 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.659211    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:38:54.678449 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.678416    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3cc201\""
Oct 31 07:38:54.680981 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.680961    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"]
Oct 31 07:38:54.691846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.691823    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4hb9\" (UniqueName: \"kubernetes.io/projected/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-kube-api-access-g4hb9\") pod \"payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq\" (UID: \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:38:54.692018 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.691992    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq\" (UID: \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:38:54.698736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.698711    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-3cc201-dockercfg-f5x2k\""
Oct 31 07:38:54.793240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.793202    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq\" (UID: \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:38:54.793429 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.793342    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4hb9\" (UniqueName: \"kubernetes.io/projected/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-kube-api-access-g4hb9\") pod \"payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq\" (UID: \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:38:54.794309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.794258    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq\" (UID: \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:38:54.813293 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.813255    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4hb9\" (UniqueName: \"kubernetes.io/projected/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-kube-api-access-g4hb9\") pod \"payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq\" (UID: \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:38:54.972588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:54.972503    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:38:55.273656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.273576    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"]
Oct 31 07:38:55.277054 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:55.277016    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb192cb88_7f38_4ea5_b71a_12ecd76fe9c4.slice/crio-4a7566914f6bf5a1050d8077e52a45f40a9e87d822fef87d8dff971e4ddb3337 WatchSource:0}: Error finding container 4a7566914f6bf5a1050d8077e52a45f40a9e87d822fef87d8dff971e4ddb3337: Status 404 returned error can't find the container with id 4a7566914f6bf5a1050d8077e52a45f40a9e87d822fef87d8dff971e4ddb3337
Oct 31 07:38:55.574749 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.574671    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"]
Oct 31 07:38:55.578422 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.578403    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:38:55.596570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.596539    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3dff3f\""
Oct 31 07:38:55.600708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.600676    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df54b1a3-b0e3-4481-b483-250b1136c78d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr\" (UID: \"df54b1a3-b0e3-4481-b483-250b1136c78d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:38:55.601025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.601004    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktb65\" (UniqueName: \"kubernetes.io/projected/df54b1a3-b0e3-4481-b483-250b1136c78d-kube-api-access-ktb65\") pod \"payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr\" (UID: \"df54b1a3-b0e3-4481-b483-250b1136c78d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:38:55.601190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.601078    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"]
Oct 31 07:38:55.691773 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.691744    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3dff3f-dockercfg-v6bpf\""
Oct 31 07:38:55.703158 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.703127    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df54b1a3-b0e3-4481-b483-250b1136c78d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr\" (UID: \"df54b1a3-b0e3-4481-b483-250b1136c78d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:38:55.703358 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.703255    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktb65\" (UniqueName: \"kubernetes.io/projected/df54b1a3-b0e3-4481-b483-250b1136c78d-kube-api-access-ktb65\") pod \"payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr\" (UID: \"df54b1a3-b0e3-4481-b483-250b1136c78d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:38:55.703997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.703973    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df54b1a3-b0e3-4481-b483-250b1136c78d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr\" (UID: \"df54b1a3-b0e3-4481-b483-250b1136c78d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:38:55.723012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.722980    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktb65\" (UniqueName: \"kubernetes.io/projected/df54b1a3-b0e3-4481-b483-250b1136c78d-kube-api-access-ktb65\") pod \"payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr\" (UID: \"df54b1a3-b0e3-4481-b483-250b1136c78d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:38:55.872745 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.872658    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq" event={"ID":"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4","Type":"ContainerStarted","Data":"ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750"}
Oct 31 07:38:55.872745 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.872736    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq" event={"ID":"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4","Type":"ContainerStarted","Data":"4a7566914f6bf5a1050d8077e52a45f40a9e87d822fef87d8dff971e4ddb3337"}
Oct 31 07:38:55.872931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.872793    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:38:55.891703 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.891677    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:38:55.916925 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:55.916874    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq" podStartSLOduration=1.916859165 podStartE2EDuration="1.916859165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:55.915860356 +0000 UTC m=+600.055020944" watchObservedRunningTime="2026-10-31 07:38:55.916859165 +0000 UTC m=+600.056019792"
Oct 31 07:38:56.067463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:56.067438    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:38:56.071703 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:56.071670    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:38:56.194452 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:56.194420    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"]
Oct 31 07:38:56.197605 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:56.197569    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf54b1a3_b0e3_4481_b483_250b1136c78d.slice/crio-24c5347a98fee057ffe48f35bd5594ca40c0ed5840321836a5a2bf72668d9990 WatchSource:0}: Error finding container 24c5347a98fee057ffe48f35bd5594ca40c0ed5840321836a5a2bf72668d9990: Status 404 returned error can't find the container with id 24c5347a98fee057ffe48f35bd5594ca40c0ed5840321836a5a2bf72668d9990
Oct 31 07:38:56.879526 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:56.879480    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr" event={"ID":"df54b1a3-b0e3-4481-b483-250b1136c78d","Type":"ContainerStarted","Data":"b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3"}
Oct 31 07:38:56.879920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:56.879554    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr" event={"ID":"df54b1a3-b0e3-4481-b483-250b1136c78d","Type":"ContainerStarted","Data":"24c5347a98fee057ffe48f35bd5594ca40c0ed5840321836a5a2bf72668d9990"}
Oct 31 07:38:56.879920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:56.879583    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:38:56.920884 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:56.920831    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr" podStartSLOduration=1.920817479 podStartE2EDuration="1.920817479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:56.917708705 +0000 UTC m=+601.056869311" watchObservedRunningTime="2026-10-31 07:38:56.920817479 +0000 UTC m=+601.059978067"
Oct 31 07:38:57.446608 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.446572    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"]
Oct 31 07:38:57.450653 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.450631    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:38:57.469036 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.469006    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"]
Oct 31 07:38:57.469386 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.469365    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-4794f5\""
Oct 31 07:38:57.479525 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.479498    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-4794f5-dockercfg-kbn6v\""
Oct 31 07:38:57.499713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.499676    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"]
Oct 31 07:38:57.504569 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.504543    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:38:57.523659 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.523630    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"]
Oct 31 07:38:57.526703 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.526677    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4t2\" (UniqueName: \"kubernetes.io/projected/538fd585-48a6-4b18-994e-61767bb14aaa-kube-api-access-tq4t2\") pod \"payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t\" (UID: \"538fd585-48a6-4b18-994e-61767bb14aaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:38:57.526816 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.526733    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538fd585-48a6-4b18-994e-61767bb14aaa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t\" (UID: \"538fd585-48a6-4b18-994e-61767bb14aaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:38:57.628008 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.627968    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4m5\" (UniqueName: \"kubernetes.io/projected/821af7a5-dc95-4425-b5b4-5709d5670e5f-kube-api-access-zb4m5\") pod \"payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r\" (UID: \"821af7a5-dc95-4425-b5b4-5709d5670e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:38:57.628172 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.628099    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821af7a5-dc95-4425-b5b4-5709d5670e5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r\" (UID: \"821af7a5-dc95-4425-b5b4-5709d5670e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:38:57.628247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.628223    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4t2\" (UniqueName: \"kubernetes.io/projected/538fd585-48a6-4b18-994e-61767bb14aaa-kube-api-access-tq4t2\") pod \"payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t\" (UID: \"538fd585-48a6-4b18-994e-61767bb14aaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:38:57.628309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.628275    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538fd585-48a6-4b18-994e-61767bb14aaa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t\" (UID: \"538fd585-48a6-4b18-994e-61767bb14aaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:38:57.628992 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.628976    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538fd585-48a6-4b18-994e-61767bb14aaa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t\" (UID: \"538fd585-48a6-4b18-994e-61767bb14aaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:38:57.647954 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.647926    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4t2\" (UniqueName: \"kubernetes.io/projected/538fd585-48a6-4b18-994e-61767bb14aaa-kube-api-access-tq4t2\") pod \"payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t\" (UID: \"538fd585-48a6-4b18-994e-61767bb14aaa\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:38:57.729257 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.729161    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4m5\" (UniqueName: \"kubernetes.io/projected/821af7a5-dc95-4425-b5b4-5709d5670e5f-kube-api-access-zb4m5\") pod \"payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r\" (UID: \"821af7a5-dc95-4425-b5b4-5709d5670e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:38:57.729257 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.729226    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821af7a5-dc95-4425-b5b4-5709d5670e5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r\" (UID: \"821af7a5-dc95-4425-b5b4-5709d5670e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:38:57.730179 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.730152    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821af7a5-dc95-4425-b5b4-5709d5670e5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r\" (UID: \"821af7a5-dc95-4425-b5b4-5709d5670e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:38:57.748803 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.748770    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4m5\" (UniqueName: \"kubernetes.io/projected/821af7a5-dc95-4425-b5b4-5709d5670e5f-kube-api-access-zb4m5\") pod \"payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r\" (UID: \"821af7a5-dc95-4425-b5b4-5709d5670e5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:38:57.764608 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.764580    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:38:57.827791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.820845    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:38:57.856405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:57.856374    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:38:58.074374 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.074235    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"]
Oct 31 07:38:58.082014 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:58.081979    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538fd585_48a6_4b18_994e_61767bb14aaa.slice/crio-9468485e50529dbf10fd9bee5cf7b409b3c640e4d6ec0d9f3be40a86277bb4a9 WatchSource:0}: Error finding container 9468485e50529dbf10fd9bee5cf7b409b3c640e4d6ec0d9f3be40a86277bb4a9: Status 404 returned error can't find the container with id 9468485e50529dbf10fd9bee5cf7b409b3c640e4d6ec0d9f3be40a86277bb4a9
Oct 31 07:38:58.147608 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.147585    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"]
Oct 31 07:38:58.150488 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:58.150457    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821af7a5_dc95_4425_b5b4_5709d5670e5f.slice/crio-bf38f9ef29f91f0adf7322508181217b9cd523277b2a3da639b20f7e556afed5 WatchSource:0}: Error finding container bf38f9ef29f91f0adf7322508181217b9cd523277b2a3da639b20f7e556afed5: Status 404 returned error can't find the container with id bf38f9ef29f91f0adf7322508181217b9cd523277b2a3da639b20f7e556afed5
Oct 31 07:38:58.894690 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.894657    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r" event={"ID":"821af7a5-dc95-4425-b5b4-5709d5670e5f","Type":"ContainerStarted","Data":"bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969"}
Oct 31 07:38:58.894875 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.894719    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r" event={"ID":"821af7a5-dc95-4425-b5b4-5709d5670e5f","Type":"ContainerStarted","Data":"bf38f9ef29f91f0adf7322508181217b9cd523277b2a3da639b20f7e556afed5"}
Oct 31 07:38:58.894875 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.894749    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:38:58.896425 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.896396    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t" event={"ID":"538fd585-48a6-4b18-994e-61767bb14aaa","Type":"ContainerStarted","Data":"be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0"}
Oct 31 07:38:58.896559 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.896442    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t" event={"ID":"538fd585-48a6-4b18-994e-61767bb14aaa","Type":"ContainerStarted","Data":"9468485e50529dbf10fd9bee5cf7b409b3c640e4d6ec0d9f3be40a86277bb4a9"}
Oct 31 07:38:58.896559 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.896548    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:38:58.940369 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.940314    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r" podStartSLOduration=1.940299148 podStartE2EDuration="1.940299148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:58.93289901 +0000 UTC m=+603.072059598" watchObservedRunningTime="2026-10-31 07:38:58.940299148 +0000 UTC m=+603.079459735"
Oct 31 07:38:58.972539 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.972500    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"]
Oct 31 07:38:58.972783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.972729    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t" podStartSLOduration=1.972713622 podStartE2EDuration="1.972713622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:58.971125676 +0000 UTC m=+603.110286266" watchObservedRunningTime="2026-10-31 07:38:58.972713622 +0000 UTC m=+603.111874211"
Oct 31 07:38:58.976529 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.976507    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:38:58.994600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.994564    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-b1fda4\""
Oct 31 07:38:58.994779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.994750    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-b1fda4-dockercfg-58b7f\""
Oct 31 07:38:58.996760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:58.996729    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"]
Oct 31 07:38:59.019542 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.019510    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"]
Oct 31 07:38:59.025791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.025769    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:38:59.041569 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.041539    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"]
Oct 31 07:38:59.045296 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.045249    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khjh8\" (UniqueName: \"kubernetes.io/projected/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-kube-api-access-khjh8\") pod \"payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726\" (UID: \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:38:59.045656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.045633    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726\" (UID: \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:38:59.147083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.146993    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khjh8\" (UniqueName: \"kubernetes.io/projected/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-kube-api-access-khjh8\") pod \"payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726\" (UID: \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:38:59.147083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.147051    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlnl\" (UniqueName: \"kubernetes.io/projected/91c0428e-b966-4e6e-8425-15bc33a040b6-kube-api-access-ddlnl\") pod \"payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f\" (UID: \"91c0428e-b966-4e6e-8425-15bc33a040b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:38:59.147535 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.147107    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c0428e-b966-4e6e-8425-15bc33a040b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f\" (UID: \"91c0428e-b966-4e6e-8425-15bc33a040b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:38:59.147535 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.147274    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726\" (UID: \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:38:59.147955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.147936    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726\" (UID: \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:38:59.166453 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.166426    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khjh8\" (UniqueName: \"kubernetes.io/projected/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-kube-api-access-khjh8\") pod \"payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726\" (UID: \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:38:59.248713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.248678    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c0428e-b966-4e6e-8425-15bc33a040b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f\" (UID: \"91c0428e-b966-4e6e-8425-15bc33a040b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:38:59.248864 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.248811    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlnl\" (UniqueName: \"kubernetes.io/projected/91c0428e-b966-4e6e-8425-15bc33a040b6-kube-api-access-ddlnl\") pod \"payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f\" (UID: \"91c0428e-b966-4e6e-8425-15bc33a040b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:38:59.249429 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.249410    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c0428e-b966-4e6e-8425-15bc33a040b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f\" (UID: \"91c0428e-b966-4e6e-8425-15bc33a040b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:38:59.268272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.268243    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlnl\" (UniqueName: \"kubernetes.io/projected/91c0428e-b966-4e6e-8425-15bc33a040b6-kube-api-access-ddlnl\") pod \"payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f\" (UID: \"91c0428e-b966-4e6e-8425-15bc33a040b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:38:59.290238 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.290216    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:38:59.339507 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.339437    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:38:59.601871 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.601840    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"]
Oct 31 07:38:59.604298 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:59.604257    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6c6f59_6834_4d62_9e21_1f263fff5b7b.slice/crio-767f41c556fe032d5c5c0ab58824b473201081905f075259407926b00a1508b6 WatchSource:0}: Error finding container 767f41c556fe032d5c5c0ab58824b473201081905f075259407926b00a1508b6: Status 404 returned error can't find the container with id 767f41c556fe032d5c5c0ab58824b473201081905f075259407926b00a1508b6
Oct 31 07:38:59.655560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.655490    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"]
Oct 31 07:38:59.659411 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:38:59.659379    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c0428e_b966_4e6e_8425_15bc33a040b6.slice/crio-f572c4253fc4a3d946cb95b487e3daf7ae5589ef9790064f74f8820932bb5cc2 WatchSource:0}: Error finding container f572c4253fc4a3d946cb95b487e3daf7ae5589ef9790064f74f8820932bb5cc2: Status 404 returned error can't find the container with id f572c4253fc4a3d946cb95b487e3daf7ae5589ef9790064f74f8820932bb5cc2
Oct 31 07:38:59.789460 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.789425    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"]
Oct 31 07:38:59.793493 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.793473    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:38:59.814861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.814836    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"]
Oct 31 07:38:59.852585 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.852550    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-648fbe\""
Oct 31 07:38:59.852779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.852750    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-648fbe-dockercfg-mmkd9\""
Oct 31 07:38:59.855551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.855515    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2s2d\" (UniqueName: \"kubernetes.io/projected/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-kube-api-access-g2s2d\") pod \"payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v\" (UID: \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:38:59.855716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.855658    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v\" (UID: \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:38:59.903684 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.903651    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f" event={"ID":"91c0428e-b966-4e6e-8425-15bc33a040b6","Type":"ContainerStarted","Data":"179a509c109305b296d36c07e45dd5fe48277e0ff7775fba63ff7d076bad7f47"}
Oct 31 07:38:59.903848 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.903712    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f" event={"ID":"91c0428e-b966-4e6e-8425-15bc33a040b6","Type":"ContainerStarted","Data":"f572c4253fc4a3d946cb95b487e3daf7ae5589ef9790064f74f8820932bb5cc2"}
Oct 31 07:38:59.903848 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.903743    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:38:59.905438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.905406    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726" event={"ID":"8f6c6f59-6834-4d62-9e21-1f263fff5b7b","Type":"ContainerStarted","Data":"f78ba7606153235b5125ad56dea27f82241bc7bb195332807e8048cf797cc5b1"}
Oct 31 07:38:59.905596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.905461    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726" event={"ID":"8f6c6f59-6834-4d62-9e21-1f263fff5b7b","Type":"ContainerStarted","Data":"767f41c556fe032d5c5c0ab58824b473201081905f075259407926b00a1508b6"}
Oct 31 07:38:59.905672 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.905652    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:38:59.945121 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.945069    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f" podStartSLOduration=1.945053411 podStartE2EDuration="1.945053411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:59.943155376 +0000 UTC m=+604.082315964" watchObservedRunningTime="2026-10-31 07:38:59.945053411 +0000 UTC m=+604.084213998"
Oct 31 07:38:59.957049 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.957012    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2s2d\" (UniqueName: \"kubernetes.io/projected/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-kube-api-access-g2s2d\") pod \"payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v\" (UID: \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:38:59.957253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.957232    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v\" (UID: \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:38:59.958242 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.958218    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v\" (UID: \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:38:59.976243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.976207    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2s2d\" (UniqueName: \"kubernetes.io/projected/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-kube-api-access-g2s2d\") pod \"payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v\" (UID: \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:38:59.982935 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:38:59.982869    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726" podStartSLOduration=1.9828470870000001 podStartE2EDuration="1.982847087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:59.980974643 +0000 UTC m=+604.120135230" watchObservedRunningTime="2026-10-31 07:38:59.982847087 +0000 UTC m=+604.122007677"
Oct 31 07:39:00.107118 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:00.107082    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:39:00.406889 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:00.406671    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"]
Oct 31 07:39:00.409514 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:39:00.409457    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0375cdc_9b5d_468c_8e5f_f989dbc5b5a8.slice/crio-13fe9412c71acc8a0fff0fed1edb123f0ce88449643d0ffccbdbe0eb5cb7d4fd WatchSource:0}: Error finding container 13fe9412c71acc8a0fff0fed1edb123f0ce88449643d0ffccbdbe0eb5cb7d4fd: Status 404 returned error can't find the container with id 13fe9412c71acc8a0fff0fed1edb123f0ce88449643d0ffccbdbe0eb5cb7d4fd
Oct 31 07:39:00.913005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:00.912968    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v" event={"ID":"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8","Type":"ContainerStarted","Data":"f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80"}
Oct 31 07:39:00.913214 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:00.913033    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v" event={"ID":"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8","Type":"ContainerStarted","Data":"13fe9412c71acc8a0fff0fed1edb123f0ce88449643d0ffccbdbe0eb5cb7d4fd"}
Oct 31 07:39:00.913418 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:00.913396    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:39:00.954046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:00.953980    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v" podStartSLOduration=1.953960925 podStartE2EDuration="1.953960925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:39:00.95290884 +0000 UTC m=+605.092069428" watchObservedRunningTime="2026-10-31 07:39:00.953960925 +0000 UTC m=+605.093121513"
Oct 31 07:39:02.061271 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:02.061235    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:39:04.070317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.070245    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"]
Oct 31 07:39:04.075184 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.075160    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:04.095473 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.095441    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"]
Oct 31 07:39:04.152681 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.152648    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-b4e8fd-dockercfg-4k84v\""
Oct 31 07:39:04.160032 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.159925    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-b4e8fd\""
Oct 31 07:39:04.260042 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.260009    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d626ba-700d-40f2-9aee-4edc15c80b85-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg\" (UID: \"81d626ba-700d-40f2-9aee-4edc15c80b85\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:04.260251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.260089    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fxb\" (UniqueName: \"kubernetes.io/projected/81d626ba-700d-40f2-9aee-4edc15c80b85-kube-api-access-g8fxb\") pod \"payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg\" (UID: \"81d626ba-700d-40f2-9aee-4edc15c80b85\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:04.360920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.360828    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d626ba-700d-40f2-9aee-4edc15c80b85-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg\" (UID: \"81d626ba-700d-40f2-9aee-4edc15c80b85\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:04.360920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.360913    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fxb\" (UniqueName: \"kubernetes.io/projected/81d626ba-700d-40f2-9aee-4edc15c80b85-kube-api-access-g8fxb\") pod \"payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg\" (UID: \"81d626ba-700d-40f2-9aee-4edc15c80b85\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:04.361850 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.361829    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d626ba-700d-40f2-9aee-4edc15c80b85-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg\" (UID: \"81d626ba-700d-40f2-9aee-4edc15c80b85\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:04.380345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.380318    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fxb\" (UniqueName: \"kubernetes.io/projected/81d626ba-700d-40f2-9aee-4edc15c80b85-kube-api-access-g8fxb\") pod \"payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg\" (UID: \"81d626ba-700d-40f2-9aee-4edc15c80b85\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:04.389627 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.389597    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:04.749303 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.749024    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"]
Oct 31 07:39:04.759426 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:39:04.759327    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d626ba_700d_40f2_9aee_4edc15c80b85.slice/crio-797c235cc25ee9aaffd7feada84052b9c446398167aeb2867da392f11e080f0b WatchSource:0}: Error finding container 797c235cc25ee9aaffd7feada84052b9c446398167aeb2867da392f11e080f0b: Status 404 returned error can't find the container with id 797c235cc25ee9aaffd7feada84052b9c446398167aeb2867da392f11e080f0b
Oct 31 07:39:04.867704 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.867679    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:39:04.868551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.868533    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:39:04.970602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.970566    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg" event={"ID":"81d626ba-700d-40f2-9aee-4edc15c80b85","Type":"ContainerStarted","Data":"38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172"}
Oct 31 07:39:04.970825 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.970809    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg" event={"ID":"81d626ba-700d-40f2-9aee-4edc15c80b85","Type":"ContainerStarted","Data":"797c235cc25ee9aaffd7feada84052b9c446398167aeb2867da392f11e080f0b"}
Oct 31 07:39:04.970912 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:04.970904    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:06.828302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:06.828010    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg" podStartSLOduration=2.827981164 podStartE2EDuration="2.827981164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:39:05.034538856 +0000 UTC m=+609.173699440" watchObservedRunningTime="2026-10-31 07:39:06.827981164 +0000 UTC m=+610.967141753"
Oct 31 07:39:06.841307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:06.829211    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:39:06.882916 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:06.882888    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:39:07.889649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:07.889617    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:39:09.908715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:09.908681    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:39:09.909714 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:09.909686    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:39:10.916107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:10.916078    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:39:10.916648 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:10.916148    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:39:11.923589 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:11.923524    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:39:12.021213 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:12.021176    5829 generic.go:356] "Generic (PLEG): container finished" podID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerID="1b806d8aaf8d2d23f88775832188ac68d5fc3995012195c09a3f010502f1f8d7" exitCode=0
Oct 31 07:39:12.021418 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:12.021214    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" event={"ID":"ef061dbf-d524-4e2b-a234-f14abfd09111","Type":"ContainerDied","Data":"1b806d8aaf8d2d23f88775832188ac68d5fc3995012195c09a3f010502f1f8d7"}
Oct 31 07:39:12.021729 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:12.021710    5829 scope.go:122] "RemoveContainer" containerID="1b806d8aaf8d2d23f88775832188ac68d5fc3995012195c09a3f010502f1f8d7"
Oct 31 07:39:12.023147 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:12.023118    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:39:12.023242 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:12.023191    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:39:12.023242 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:12.023229    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:39:13.033475 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.033431    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" event={"ID":"ef061dbf-d524-4e2b-a234-f14abfd09111","Type":"ContainerStarted","Data":"c2dd6256bcd4746fe4b1311ce9a4dd3b5197803a73e2b194f426eaa93c01da8d"}
Oct 31 07:39:13.035342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.035313    5829 generic.go:356] "Generic (PLEG): container finished" podID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerID="4ab8a0adf6b0a59509566d296e59ce0e87751018e803a6cf57b5723a48ccdd39" exitCode=0
Oct 31 07:39:13.035474 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.035380    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" event={"ID":"d130a1c7-6b3f-4168-af6a-451147ed223d","Type":"ContainerDied","Data":"4ab8a0adf6b0a59509566d296e59ce0e87751018e803a6cf57b5723a48ccdd39"}
Oct 31 07:39:13.035722 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.035708    5829 scope.go:122] "RemoveContainer" containerID="4ab8a0adf6b0a59509566d296e59ce0e87751018e803a6cf57b5723a48ccdd39"
Oct 31 07:39:13.036996 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.036963    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:39:13.037095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.037047    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:39:13.037157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.037094    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:39:13.448983 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.448945    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:39:13.449237 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.449214    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 07:39:13.449327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:13.449264    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 07:39:14.060379 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:14.060341    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" event={"ID":"d130a1c7-6b3f-4168-af6a-451147ed223d","Type":"ContainerStarted","Data":"2f6fad6bf73af5c29a2dd7843f9438d618605570f3bfb73dbb16b28d7c254dfc"}
Oct 31 07:39:14.094229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:14.094195    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:39:14.094467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:14.094436    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 07:39:14.094513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:14.094491    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 07:39:14.453329 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:14.449704    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 07:39:14.453329 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:14.449770    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 07:39:15.094180 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:15.094137    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 07:39:15.094710 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:15.094215    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 07:39:15.451314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:15.450823    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 07:39:15.451314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:15.450903    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 07:39:15.980303 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:15.979507    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:39:16.094629 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:16.094591    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 07:39:16.095022 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:16.094644    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 07:39:16.453335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:16.449735    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 07:39:16.453335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:16.449802    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 07:39:16.565901 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:39:16.565863    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:39:17.094765 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:17.094728    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 07:39:17.095152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:17.094783    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 07:39:17.463728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:17.463691    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 31 07:39:17.463901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:17.463753    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 31 07:39:18.095010 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:18.094973    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 07:39:18.095445 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:18.095027    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 07:39:18.453856 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:18.453822    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:39:18.454309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:18.454262    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:39:18.454883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:18.454860    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:39:18.899149 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:18.899116    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:39:19.094234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:19.094200    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 31 07:39:19.094409 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:19.094255    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 31 07:39:19.833558 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:19.833529    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:39:20.098896 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:20.098819    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:39:20.099949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:20.099924    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:39:23.889789 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:23.889755    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"]
Oct 31 07:39:23.895064 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:23.895041    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:23.917267 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:23.917236    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"]
Oct 31 07:39:23.926979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:23.926946    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1fda4\"/\"simulator-w5-b1fda4-kserve-self-signed-certs\""
Oct 31 07:39:23.927194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:23.926986    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1fda4\"/\"default-dockercfg-dzpd4\""
Oct 31 07:39:23.927339 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:23.926992    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b1fda4\"/\"openshift-service-ca.crt\""
Oct 31 07:39:23.927427 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:23.927113    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-b1fda4\"/\"kube-root-ca.crt\""
Oct 31 07:39:24.040577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.040539    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-home\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.040771 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.040595    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70adf136-b28a-439e-b53f-afce743b4a79-tls-certs\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.040771 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.040619    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fcbt\" (UniqueName: \"kubernetes.io/projected/70adf136-b28a-439e-b53f-afce743b4a79-kube-api-access-8fcbt\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.040771 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.040645    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-model-cache\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.040892 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.040782    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-dshm\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.040892 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.040830    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-tmp-dir\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.142002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.141885    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-tmp-dir\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.142187 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.142057    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-home\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.142187 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.142116    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70adf136-b28a-439e-b53f-afce743b4a79-tls-certs\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.142187 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.142153    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fcbt\" (UniqueName: \"kubernetes.io/projected/70adf136-b28a-439e-b53f-afce743b4a79-kube-api-access-8fcbt\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.142382 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.142192    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-model-cache\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.142382 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.142247    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-dshm\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.142533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.142499    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-tmp-dir\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.142790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.142727    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-model-cache\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.143442 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.143418    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-home\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.145117 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.145090    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-dshm\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.145989 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.145965    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70adf136-b28a-439e-b53f-afce743b4a79-tls-certs\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.162488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.162459    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fcbt\" (UniqueName: \"kubernetes.io/projected/70adf136-b28a-439e-b53f-afce743b4a79-kube-api-access-8fcbt\") pod \"simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") " pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.208219 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.208184    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:24.512028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:24.511993    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"]
Oct 31 07:39:24.514450 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:39:24.514419    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70adf136_b28a_439e_b53f_afce743b4a79.slice/crio-ecdc62cdd0630ec9e464b14cd50902eb6863bba36e573d56be62443f15c35102 WatchSource:0}: Error finding container ecdc62cdd0630ec9e464b14cd50902eb6863bba36e573d56be62443f15c35102: Status 404 returned error can't find the container with id ecdc62cdd0630ec9e464b14cd50902eb6863bba36e573d56be62443f15c35102
Oct 31 07:39:25.129551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:25.129512    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" event={"ID":"70adf136-b28a-439e-b53f-afce743b4a79","Type":"ContainerStarted","Data":"4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7"}
Oct 31 07:39:25.129952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:25.129563    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" event={"ID":"70adf136-b28a-439e-b53f-afce743b4a79","Type":"ContainerStarted","Data":"ecdc62cdd0630ec9e464b14cd50902eb6863bba36e573d56be62443f15c35102"}
Oct 31 07:39:25.171954 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:25.171899    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podStartSLOduration=2.171882979 podStartE2EDuration="2.171882979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:39:25.169307399 +0000 UTC m=+629.308467979" watchObservedRunningTime="2026-10-31 07:39:25.171882979 +0000 UTC m=+629.311043577"
Oct 31 07:39:25.209201 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:25.209153    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:27.559868 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:27.559837    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:39:34.209236 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:34.209203    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:34.222107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:34.222076    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:35.198191 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:35.198158    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:39:39.576198 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:39.576114    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:39:41.232314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.232262    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"]
Oct 31 07:39:41.237148 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.237121    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.242767 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.242732    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"]
Oct 31 07:39:41.252946 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.252916    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b4e8fd\"/\"kube-root-ca.crt\""
Oct 31 07:39:41.253098 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.252981    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b4e8fd\"/\"simulator-w2-b4e8fd-kserve-self-signed-certs\""
Oct 31 07:39:41.265526 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.265499    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-b4e8fd\"/\"openshift-service-ca.crt\""
Oct 31 07:39:41.265656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.265592    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-b4e8fd\"/\"default-dockercfg-hlg77\""
Oct 31 07:39:41.332454 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.332421    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tls-certs\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.332640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.332493    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-model-cache\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.332640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.332598    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-home\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.332640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.332632    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tmp-dir\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.332775 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.332663    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6kq7\" (UniqueName: \"kubernetes.io/projected/91d1cb28-6264-4ac6-9a1b-5c19788c262c-kube-api-access-l6kq7\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.332775 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.332750    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-dshm\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.434144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.434099    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tls-certs\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.434383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.434240    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-model-cache\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.434383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.434378    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-home\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.434510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.434412    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tmp-dir\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.434510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.434447    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6kq7\" (UniqueName: \"kubernetes.io/projected/91d1cb28-6264-4ac6-9a1b-5c19788c262c-kube-api-access-l6kq7\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.434510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.434484    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-dshm\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.434739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.434712    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-model-cache\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.435395 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.434927    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tmp-dir\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.435395 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.435178    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-home\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.437035 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.437010    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-dshm\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.437035 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.437023    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tls-certs\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.454584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.454550    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6kq7\" (UniqueName: \"kubernetes.io/projected/91d1cb28-6264-4ac6-9a1b-5c19788c262c-kube-api-access-l6kq7\") pod \"simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") " pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.552046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.551949    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:41.858973 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:41.858943    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"]
Oct 31 07:39:41.861155 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:39:41.861123    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91d1cb28_6264_4ac6_9a1b_5c19788c262c.slice/crio-570481758b97439d1eb6c332c85fe7ceb0444fd80756cc52de7934846af716e0 WatchSource:0}: Error finding container 570481758b97439d1eb6c332c85fe7ceb0444fd80756cc52de7934846af716e0: Status 404 returned error can't find the container with id 570481758b97439d1eb6c332c85fe7ceb0444fd80756cc52de7934846af716e0
Oct 31 07:39:42.242634 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:42.242601    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" event={"ID":"91d1cb28-6264-4ac6-9a1b-5c19788c262c","Type":"ContainerStarted","Data":"0961a92277fa1445be4d47c3917d46cbc7793ede11257f8fd4431f16a0faefe1"}
Oct 31 07:39:42.243072 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:42.242644    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" event={"ID":"91d1cb28-6264-4ac6-9a1b-5c19788c262c","Type":"ContainerStarted","Data":"570481758b97439d1eb6c332c85fe7ceb0444fd80756cc52de7934846af716e0"}
Oct 31 07:39:42.282374 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:42.282308    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podStartSLOduration=1.282265701 podStartE2EDuration="1.282265701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:39:42.282079073 +0000 UTC m=+646.421239687" watchObservedRunningTime="2026-10-31 07:39:42.282265701 +0000 UTC m=+646.421426285"
Oct 31 07:39:42.552144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:42.552049    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:46.511652 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:39:46.511613    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:39:49.102777 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.102740    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"]
Oct 31 07:39:49.107228 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.107200    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.135897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.135866    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3cc201\"/\"openshift-service-ca.crt\""
Oct 31 07:39:49.136073 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.135944    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3cc201\"/\"kube-root-ca.crt\""
Oct 31 07:39:49.136073 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.135983    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc201\"/\"premium-w3-3cc201-kserve-self-signed-certs\""
Oct 31 07:39:49.136073 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.136050    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc201\"/\"default-dockercfg-j99g8\""
Oct 31 07:39:49.138148 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.138123    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"]
Oct 31 07:39:49.223648 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.223609    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-dshm\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.223826 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.223723    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tls-certs\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.223826 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.223760    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-model-cache\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.223911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.223832    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tmp-dir\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.223911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.223885    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-home\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.223981 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.223913    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcg4t\" (UniqueName: \"kubernetes.io/projected/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-kube-api-access-xcg4t\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.324711 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.324662    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tls-certs\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.324906 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.324845    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-model-cache\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.324978 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.324961    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tmp-dir\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.325052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.325035    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-home\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.325109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.325084    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcg4t\" (UniqueName: \"kubernetes.io/projected/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-kube-api-access-xcg4t\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.325161 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.325126    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-model-cache\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.325243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.325222    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-dshm\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.325518 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.325406    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tmp-dir\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.328485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.325636    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-home\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.328485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.328047    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-dshm\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.331608 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.331584    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tls-certs\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.345178 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.345145    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcg4t\" (UniqueName: \"kubernetes.io/projected/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-kube-api-access-xcg4t\") pod \"premium-w3-3cc201-kserve-669c9cd68c-wmhxt\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") " pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.422724 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.422686    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:49.723201 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:49.723178    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"]
Oct 31 07:39:49.725617 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:39:49.725564    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b33107c_7771_44a0_8600_4ab5d2b8d1e4.slice/crio-bc4a87d3d3f4453f5009492952f0f2f7458c7917c119272d42e3433c48be3886 WatchSource:0}: Error finding container bc4a87d3d3f4453f5009492952f0f2f7458c7917c119272d42e3433c48be3886: Status 404 returned error can't find the container with id bc4a87d3d3f4453f5009492952f0f2f7458c7917c119272d42e3433c48be3886
Oct 31 07:39:50.291045 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:50.291003    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" event={"ID":"4b33107c-7771-44a0-8600-4ab5d2b8d1e4","Type":"ContainerStarted","Data":"4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1"}
Oct 31 07:39:50.291458 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:50.291055    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" event={"ID":"4b33107c-7771-44a0-8600-4ab5d2b8d1e4","Type":"ContainerStarted","Data":"bc4a87d3d3f4453f5009492952f0f2f7458c7917c119272d42e3433c48be3886"}
Oct 31 07:39:50.331624 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:50.331563    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podStartSLOduration=1.331544734 podStartE2EDuration="1.331544734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:39:50.329164978 +0000 UTC m=+654.468325568" watchObservedRunningTime="2026-10-31 07:39:50.331544734 +0000 UTC m=+654.470705324"
Oct 31 07:39:50.423476 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:50.423440    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:51.552440 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:51.552398    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:51.564848 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:51.564821    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:52.273769 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:52.273737    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:39:52.313342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:52.313318    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:39:59.423355 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:59.423315    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:39:59.436089 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:39:59.436059    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:40:00.361479 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:00.361449    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:40:01.543004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.542970    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"]
Oct 31 07:40:01.548461 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.548437    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.565862 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.565834    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"]
Oct 31 07:40:01.576144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.576115    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-b1fda4\"/\"premium-w5-b1fda4-kserve-self-signed-certs\""
Oct 31 07:40:01.665485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.665433    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-tmp-dir\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.665678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.665565    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68hm2\" (UniqueName: \"kubernetes.io/projected/dc39e212-ced0-4425-a28e-e38ebfc09652-kube-api-access-68hm2\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.665678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.665617    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-home\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.665678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.665647    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39e212-ced0-4425-a28e-e38ebfc09652-tls-certs\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.665779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.665743    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-model-cache\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.665857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.665796    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-dshm\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.766650 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.766607    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-tmp-dir\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.766846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.766707    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68hm2\" (UniqueName: \"kubernetes.io/projected/dc39e212-ced0-4425-a28e-e38ebfc09652-kube-api-access-68hm2\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.766846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.766753    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-home\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.766846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.766791    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39e212-ced0-4425-a28e-e38ebfc09652-tls-certs\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.766846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.766841    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-model-cache\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.767062 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.766930    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-dshm\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.767062 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.766970    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-tmp-dir\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.767171 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.767145    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-home\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.767234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.767174    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-model-cache\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.769050 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.769029    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-dshm\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.769369 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.769352    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39e212-ced0-4425-a28e-e38ebfc09652-tls-certs\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.786160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.786141    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68hm2\" (UniqueName: \"kubernetes.io/projected/dc39e212-ced0-4425-a28e-e38ebfc09652-kube-api-access-68hm2\") pod \"premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") " pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:01.864028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:01.863936    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:02.172774 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:02.172743    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"]
Oct 31 07:40:02.174535 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:40:02.174500    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc39e212_ced0_4425_a28e_e38ebfc09652.slice/crio-11aaf94606c340d7e694342bca6f6205e62a6148794d455f11b229c07d677bdb WatchSource:0}: Error finding container 11aaf94606c340d7e694342bca6f6205e62a6148794d455f11b229c07d677bdb: Status 404 returned error can't find the container with id 11aaf94606c340d7e694342bca6f6205e62a6148794d455f11b229c07d677bdb
Oct 31 07:40:02.374122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:02.374082    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" event={"ID":"dc39e212-ced0-4425-a28e-e38ebfc09652","Type":"ContainerStarted","Data":"19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24"}
Oct 31 07:40:02.374122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:02.374128    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" event={"ID":"dc39e212-ced0-4425-a28e-e38ebfc09652","Type":"ContainerStarted","Data":"11aaf94606c340d7e694342bca6f6205e62a6148794d455f11b229c07d677bdb"}
Oct 31 07:40:02.413965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:02.413629    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podStartSLOduration=1.413615697 podStartE2EDuration="1.413615697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:40:02.41252907 +0000 UTC m=+666.551689657" watchObservedRunningTime="2026-10-31 07:40:02.413615697 +0000 UTC m=+666.552776285"
Oct 31 07:40:02.864830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:02.864797    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:06.649231 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:06.649194    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:40:11.864577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:11.864543    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:11.877252 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:11.877226    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:12.446040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:12.446003    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:40:16.510175 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:40:16.510132    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:40:20.518084 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:20.518047    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:40:21.121047 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.121010    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"]
Oct 31 07:40:21.126727 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.126697    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.144891 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.144862    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"]
Oct 31 07:40:21.155065 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.155037    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3dff3f\"/\"openshift-service-ca.crt\""
Oct 31 07:40:21.155246 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.155079    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3dff3f\"/\"default-dockercfg-dztn4\""
Oct 31 07:40:21.155246 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.155100    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3dff3f\"/\"premium-w4-3dff3f-kserve-self-signed-certs\""
Oct 31 07:40:21.155246 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.155106    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3dff3f\"/\"kube-root-ca.crt\""
Oct 31 07:40:21.291914 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.291879    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsp65\" (UniqueName: \"kubernetes.io/projected/549cac55-429e-46a1-b33d-a619f959a170-kube-api-access-qsp65\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.292125 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.292016    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-dshm\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.292125 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.292049    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549cac55-429e-46a1-b33d-a619f959a170-tls-certs\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.292252 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.292152    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-tmp-dir\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.292346 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.292248    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-home\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.292405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.292387    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-model-cache\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.393047 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.393005    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsp65\" (UniqueName: \"kubernetes.io/projected/549cac55-429e-46a1-b33d-a619f959a170-kube-api-access-qsp65\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.393251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.393079    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-dshm\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.393251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.393099    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549cac55-429e-46a1-b33d-a619f959a170-tls-certs\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.393251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.393123    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-tmp-dir\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.393251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.393164    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-home\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.393251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.393242    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-model-cache\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.393638 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.393611    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-tmp-dir\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.393761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.393647    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-model-cache\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.393761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.393723    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-home\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.395564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.395543    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-dshm\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.395673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.395642    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549cac55-429e-46a1-b33d-a619f959a170-tls-certs\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.412867 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.412838    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsp65\" (UniqueName: \"kubernetes.io/projected/549cac55-429e-46a1-b33d-a619f959a170-kube-api-access-qsp65\") pod \"premium-w4-3dff3f-kserve-68fbd694dd-q7vlv\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") " pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.440724 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.440693    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:21.745482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:21.745457    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"]
Oct 31 07:40:21.747836 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:40:21.747804    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod549cac55_429e_46a1_b33d_a619f959a170.slice/crio-a0444008303d41cf58ba5f1839c5df5288cb1b4202d15b35ee73db686c85c610 WatchSource:0}: Error finding container a0444008303d41cf58ba5f1839c5df5288cb1b4202d15b35ee73db686c85c610: Status 404 returned error can't find the container with id a0444008303d41cf58ba5f1839c5df5288cb1b4202d15b35ee73db686c85c610
Oct 31 07:40:22.508372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:22.508330    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" event={"ID":"549cac55-429e-46a1-b33d-a619f959a170","Type":"ContainerStarted","Data":"ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650"}
Oct 31 07:40:22.508557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:22.508382    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" event={"ID":"549cac55-429e-46a1-b33d-a619f959a170","Type":"ContainerStarted","Data":"a0444008303d41cf58ba5f1839c5df5288cb1b4202d15b35ee73db686c85c610"}
Oct 31 07:40:22.549918 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:22.549866    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podStartSLOduration=1.549851272 podStartE2EDuration="1.549851272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:40:22.547450978 +0000 UTC m=+686.686611565" watchObservedRunningTime="2026-10-31 07:40:22.549851272 +0000 UTC m=+686.689011896"
Oct 31 07:40:23.441035 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:23.441006    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:31.441534 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:31.441495    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:31.455848 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:31.455816    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:31.574839 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:31.574804    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:40:38.850797 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.850761    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"]
Oct 31 07:40:38.856144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.856122    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:38.873701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.873672    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"]
Oct 31 07:40:38.883666 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.883639    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:40:38.883830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.883674    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:40:38.883830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.883700    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:40:38.884523 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.884500    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:40:38.995198 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.995151    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7748bfb0-dzl64\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:38.995427 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.995210    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7748bfb0-dzl64\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:38.995427 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:38.995371    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrx8x\" (UniqueName: \"kubernetes.io/projected/34a518f8-3868-43e6-a909-39ab75e94918-kube-api-access-vrx8x\") pod \"maas-api-revoke-sub-7748bfb0-dzl64\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:39.096225 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:39.096190    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7748bfb0-dzl64\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:39.096441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:39.096241    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7748bfb0-dzl64\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:39.096441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:39.096264    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrx8x\" (UniqueName: \"kubernetes.io/projected/34a518f8-3868-43e6-a909-39ab75e94918-kube-api-access-vrx8x\") pod \"maas-api-revoke-sub-7748bfb0-dzl64\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:39.096887 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:39.096865    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7748bfb0-dzl64\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:39.098587 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:39.098553    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7748bfb0-dzl64\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:39.117179 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:39.117107    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrx8x\" (UniqueName: \"kubernetes.io/projected/34a518f8-3868-43e6-a909-39ab75e94918-kube-api-access-vrx8x\") pod \"maas-api-revoke-sub-7748bfb0-dzl64\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:39.170941 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:39.170904    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:39.472210 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:39.472187    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"]
Oct 31 07:40:39.475008 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:40:39.474978    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a518f8_3868_43e6_a909_39ab75e94918.slice/crio-3e1a5b297bb1f2bbeae0bb7870f599a768f63fd405fd781298845e87f6606082 WatchSource:0}: Error finding container 3e1a5b297bb1f2bbeae0bb7870f599a768f63fd405fd781298845e87f6606082: Status 404 returned error can't find the container with id 3e1a5b297bb1f2bbeae0bb7870f599a768f63fd405fd781298845e87f6606082
Oct 31 07:40:39.622208 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:39.622164    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64" event={"ID":"34a518f8-3868-43e6-a909-39ab75e94918","Type":"ContainerStarted","Data":"3e1a5b297bb1f2bbeae0bb7870f599a768f63fd405fd781298845e87f6606082"}
Oct 31 07:40:42.359508 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:42.359471    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:40:42.643305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:42.643245    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64" event={"ID":"34a518f8-3868-43e6-a909-39ab75e94918","Type":"ContainerStarted","Data":"7d88d36b0c3af9e61447ee13488ba9730c71ae20d1a01305dae24a54fe2d754d"}
Oct 31 07:40:42.686708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:42.686642    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64" podStartSLOduration=1.991561274 podStartE2EDuration="4.686621573s" totalImagesPullingTime="2.695060299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:40:42.682871055 +0000 UTC m=+706.822031644" watchObservedRunningTime="2026-10-31 07:40:42.686621573 +0000 UTC m=+706.825782162"
Oct 31 07:40:43.650515 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:43.650473    5829 generic.go:356] "Generic (PLEG): container finished" podID="34a518f8-3868-43e6-a909-39ab75e94918" containerID="7d88d36b0c3af9e61447ee13488ba9730c71ae20d1a01305dae24a54fe2d754d" exitCode=0
Oct 31 07:40:43.650915 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:43.650554    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64" event={"ID":"34a518f8-3868-43e6-a909-39ab75e94918","Type":"ContainerDied","Data":"7d88d36b0c3af9e61447ee13488ba9730c71ae20d1a01305dae24a54fe2d754d"}
Oct 31 07:40:44.792833 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.792808    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:44.865661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.865628    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-openshift-ca\") pod \"34a518f8-3868-43e6-a909-39ab75e94918\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") "
Oct 31 07:40:44.865859 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.865745    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a518f8-3868-43e6-a909-39ab75e94918-kube-api-access-vrx8x\" (UniqueName: \"kubernetes.io/projected/34a518f8-3868-43e6-a909-39ab75e94918-kube-api-access-vrx8x\") pod \"34a518f8-3868-43e6-a909-39ab75e94918\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") "
Oct 31 07:40:44.865924 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.865910    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-xks-ca\") pod \"34a518f8-3868-43e6-a909-39ab75e94918\" (UID: \"34a518f8-3868-43e6-a909-39ab75e94918\") "
Oct 31 07:40:44.866259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.866229    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-openshift-ca" pod "34a518f8-3868-43e6-a909-39ab75e94918" (UID: "34a518f8-3868-43e6-a909-39ab75e94918"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:40:44.868391 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.868344    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-xks-ca" pod "34a518f8-3868-43e6-a909-39ab75e94918" (UID: "34a518f8-3868-43e6-a909-39ab75e94918"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:40:44.868497 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.868398    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a518f8-3868-43e6-a909-39ab75e94918-kube-api-access-vrx8x" pod "34a518f8-3868-43e6-a909-39ab75e94918" (UID: "34a518f8-3868-43e6-a909-39ab75e94918"). InnerVolumeSpecName "kube-api-access-vrx8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:40:44.967632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.967553    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:40:44.967632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.967580    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34a518f8-3868-43e6-a909-39ab75e94918-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:40:44.967632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:44.967589    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrx8x\" (UniqueName: \"kubernetes.io/projected/34a518f8-3868-43e6-a909-39ab75e94918-kube-api-access-vrx8x\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:40:45.663255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:45.663222    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"
Oct 31 07:40:45.663452 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:45.663252    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64" event={"ID":"34a518f8-3868-43e6-a909-39ab75e94918","Type":"ContainerDied","Data":"3e1a5b297bb1f2bbeae0bb7870f599a768f63fd405fd781298845e87f6606082"}
Oct 31 07:40:45.663452 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:45.663321    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e1a5b297bb1f2bbeae0bb7870f599a768f63fd405fd781298845e87f6606082"
Oct 31 07:40:46.511397 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:40:46.511358    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:40:48.348867 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.348830    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"]
Oct 31 07:40:48.349477 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.349460    5829 cpu_manager.go:414] "containerMap: removing container" podUID="34a518f8-3868-43e6-a909-39ab75e94918" containerName="revoke-keys"
Oct 31 07:40:48.349548 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.349479    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a518f8-3868-43e6-a909-39ab75e94918" containerName="revoke-keys"
Oct 31 07:40:48.349608 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.349597    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a518f8-3868-43e6-a909-39ab75e94918" containerName="revoke-keys"
Oct 31 07:40:48.353189 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.353163    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.368667 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.368633    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3dff3f-openshift-default-dockercfg-8q5cz\""
Oct 31 07:40:48.384203 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.384163    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"]
Oct 31 07:40:48.507746 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.507707    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-credential-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.507932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.507760    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-podinfo\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.507932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.507874    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-envoy\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.507932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.507927    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c075d-8c75-474b-b71b-15ab59bc2926-istiod-ca-cert\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.508101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.508031    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-certs\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.508155 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.508108    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2n4\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-kube-api-access-tb2n4\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.508217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.508197    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.508258 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.508242    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-token\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.508355 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.508339    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-data\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.609653 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.609564    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.609653 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.609616    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-token\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.609653 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.609649    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-data\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.609873 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.609695    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-credential-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.609873 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.609833    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-podinfo\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.609976 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.609890    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-envoy\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.609976 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.609957    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c075d-8c75-474b-b71b-15ab59bc2926-istiod-ca-cert\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.610089 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.609978    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.610089 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.609989    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-certs\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.610089 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.610054    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2n4\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-kube-api-access-tb2n4\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.610089 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.610057    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-credential-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.610089 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.610072    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-data\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.610379 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.610267    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-certs\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.610652 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.610620    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c075d-8c75-474b-b71b-15ab59bc2926-istiod-ca-cert\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.612263 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.612239    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-envoy\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.612587 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.612572    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-podinfo\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.629186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.629157    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-token\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.629483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.629457    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2n4\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-kube-api-access-tb2n4\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.667357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.667316    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:48.993170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.993139    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"]
Oct 31 07:40:48.996397 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:40:48.996368    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5c075d_8c75_474b_b71b_15ab59bc2926.slice/crio-36de0b6eefada1b54f0d497b1fed8590a5a300dc07ddd61138b6e3c857b739d3 WatchSource:0}: Error finding container 36de0b6eefada1b54f0d497b1fed8590a5a300dc07ddd61138b6e3c857b739d3: Status 404 returned error can't find the container with id 36de0b6eefada1b54f0d497b1fed8590a5a300dc07ddd61138b6e3c857b739d3
Oct 31 07:40:48.998688 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.998654    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:40:48.998822 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.998761    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:40:48.998822 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:48.998796    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:40:49.697727 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:49.697678    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" event={"ID":"7b5c075d-8c75-474b-b71b-15ab59bc2926","Type":"ContainerStarted","Data":"748f613b3b4fc57bbfea03542222a043ca71d6853bed7bfc9afc03bda7812114"}
Oct 31 07:40:49.698187 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:49.697770    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" event={"ID":"7b5c075d-8c75-474b-b71b-15ab59bc2926","Type":"ContainerStarted","Data":"36de0b6eefada1b54f0d497b1fed8590a5a300dc07ddd61138b6e3c857b739d3"}
Oct 31 07:40:49.758246 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:49.758185    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" podStartSLOduration=1.758170467 podStartE2EDuration="1.758170467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:40:49.757657056 +0000 UTC m=+713.896817667" watchObservedRunningTime="2026-10-31 07:40:49.758170467 +0000 UTC m=+713.897331056"
Oct 31 07:40:50.667988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:50.667948    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:50.669860 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:50.669839    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 07:40:50.669928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:50.669898    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 07:40:51.668451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:51.668408    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 07:40:51.668875 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:51.668468    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 07:40:52.668230 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:52.668188    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 07:40:52.668444 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:52.668267    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 07:40:53.128682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.128575    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"]
Oct 31 07:40:53.133827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.133802    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.152579 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.152540    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"]
Oct 31 07:40:53.163047 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.163018    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:40:53.163240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.163086    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:40:53.163433 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.163136    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:40:53.163516 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.163237    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:40:53.175658 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.175633    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a79e028f-vb6md\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.175789 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.175677    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8gjs\" (UniqueName: \"kubernetes.io/projected/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-kube-api-access-l8gjs\") pod \"maas-api-revoke-sub-a79e028f-vb6md\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.175875 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.175851    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a79e028f-vb6md\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.276745 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.276714    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8gjs\" (UniqueName: \"kubernetes.io/projected/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-kube-api-access-l8gjs\") pod \"maas-api-revoke-sub-a79e028f-vb6md\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.277037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.277021    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a79e028f-vb6md\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.277213 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.277199    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a79e028f-vb6md\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.277894 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.277875    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a79e028f-vb6md\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.279273 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.279254    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a79e028f-vb6md\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.296075 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.296052    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8gjs\" (UniqueName: \"kubernetes.io/projected/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-kube-api-access-l8gjs\") pod \"maas-api-revoke-sub-a79e028f-vb6md\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.447496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.447465    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:53.668556 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.668518    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 07:40:53.668744 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.668572    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 07:40:53.775805 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:53.775772    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"]
Oct 31 07:40:53.778275 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:40:53.778248    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac76004_7ba5_4b5c_9fa8_a967cf0b17e8.slice/crio-76b971358b05e95635151fa7439ee8e239a82e908a6ab3abc50f328cb7e407fa WatchSource:0}: Error finding container 76b971358b05e95635151fa7439ee8e239a82e908a6ab3abc50f328cb7e407fa: Status 404 returned error can't find the container with id 76b971358b05e95635151fa7439ee8e239a82e908a6ab3abc50f328cb7e407fa
Oct 31 07:40:54.668460 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:54.668418    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 31 07:40:54.668871 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:54.668477    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 31 07:40:54.753060 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:54.753005    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md" event={"ID":"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8","Type":"ContainerStarted","Data":"cdaca30d5ecc5de2cc8b735e2ac34998f9a0ddfb8c92da424339ad72cc43118f"}
Oct 31 07:40:54.753383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:54.753360    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md" event={"ID":"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8","Type":"ContainerStarted","Data":"76b971358b05e95635151fa7439ee8e239a82e908a6ab3abc50f328cb7e407fa"}
Oct 31 07:40:54.794077 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:54.794022    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md" podStartSLOduration=1.794009209 podStartE2EDuration="1.794009209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:40:54.791622586 +0000 UTC m=+718.930783174" watchObservedRunningTime="2026-10-31 07:40:54.794009209 +0000 UTC m=+718.933169817"
Oct 31 07:40:55.671909 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:55.671878    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:55.672319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:55.672175    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:55.673012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:55.672995    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:40:55.761517 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:55.761487    5829 generic.go:356] "Generic (PLEG): container finished" podID="9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8" containerID="cdaca30d5ecc5de2cc8b735e2ac34998f9a0ddfb8c92da424339ad72cc43118f" exitCode=0
Oct 31 07:40:55.761695 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:55.761572    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md" event={"ID":"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8","Type":"ContainerDied","Data":"cdaca30d5ecc5de2cc8b735e2ac34998f9a0ddfb8c92da424339ad72cc43118f"}
Oct 31 07:40:56.910492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:56.910464    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:57.023222 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.023132    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-kube-api-access-l8gjs\" (UniqueName: \"kubernetes.io/projected/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-kube-api-access-l8gjs\") pod \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") "
Oct 31 07:40:57.023403 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.023313    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-openshift-ca\") pod \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") "
Oct 31 07:40:57.023403 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.023348    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-xks-ca\") pod \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\" (UID: \"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8\") "
Oct 31 07:40:57.023668 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.023635    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-openshift-ca" pod "9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8" (UID: "9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:40:57.025321 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.025273    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-xks-ca" pod "9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8" (UID: "9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:40:57.025321 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.025303    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-kube-api-access-l8gjs" pod "9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8" (UID: "9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8"). InnerVolumeSpecName "kube-api-access-l8gjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:40:57.124734 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.124695    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:40:57.124734 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.124728    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:40:57.124734 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.124737    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8gjs\" (UniqueName: \"kubernetes.io/projected/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8-kube-api-access-l8gjs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:40:57.776332 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.776305    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"
Oct 31 07:40:57.776332 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.776313    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md" event={"ID":"9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8","Type":"ContainerDied","Data":"76b971358b05e95635151fa7439ee8e239a82e908a6ab3abc50f328cb7e407fa"}
Oct 31 07:40:57.776530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:57.776390    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76b971358b05e95635151fa7439ee8e239a82e908a6ab3abc50f328cb7e407fa"
Oct 31 07:40:58.649364 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.649329    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"]
Oct 31 07:40:58.649920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.649906    5829 cpu_manager.go:414] "containerMap: removing container" podUID="9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8" containerName="revoke-keys"
Oct 31 07:40:58.649960 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.649921    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8" containerName="revoke-keys"
Oct 31 07:40:58.650046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.650036    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8" containerName="revoke-keys"
Oct 31 07:40:58.653623 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.653603    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.669572 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.669547    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:40:58.682683 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.682651    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:40:58.682683 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.682674    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:40:58.682919 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.682702    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:40:58.685427 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.685372    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"]
Oct 31 07:40:58.745608 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.745572    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x685j\" (UniqueName: \"kubernetes.io/projected/de922643-08e7-4c2a-9e9a-65a67f902d10-kube-api-access-x685j\") pod \"maas-api-revoke-sub-137733a5-xrrnc\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.745759 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.745622    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-137733a5-xrrnc\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.745759 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.745734    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-137733a5-xrrnc\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.846595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.846559    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x685j\" (UniqueName: \"kubernetes.io/projected/de922643-08e7-4c2a-9e9a-65a67f902d10-kube-api-access-x685j\") pod \"maas-api-revoke-sub-137733a5-xrrnc\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.846800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.846607    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-137733a5-xrrnc\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.846800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.846657    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-137733a5-xrrnc\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.847463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.847443    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-137733a5-xrrnc\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.849043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.849024    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-137733a5-xrrnc\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.866123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.866097    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x685j\" (UniqueName: \"kubernetes.io/projected/de922643-08e7-4c2a-9e9a-65a67f902d10-kube-api-access-x685j\") pod \"maas-api-revoke-sub-137733a5-xrrnc\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:58.967433 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:58.967346    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:40:59.275950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:59.275922    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"]
Oct 31 07:40:59.277994 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:40:59.277965    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde922643_08e7_4c2a_9e9a_65a67f902d10.slice/crio-c747f93c48030481ecedcdfdeb5468ccc63fed5211fc53fbad9f1edab7ae0e1f WatchSource:0}: Error finding container c747f93c48030481ecedcdfdeb5468ccc63fed5211fc53fbad9f1edab7ae0e1f: Status 404 returned error can't find the container with id c747f93c48030481ecedcdfdeb5468ccc63fed5211fc53fbad9f1edab7ae0e1f
Oct 31 07:40:59.789911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:59.789864    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc" event={"ID":"de922643-08e7-4c2a-9e9a-65a67f902d10","Type":"ContainerStarted","Data":"0b1f14cdabb9ff7844b7fb9e434d344dd48aff5d2cd32c46e4919d0aaca847bf"}
Oct 31 07:40:59.790307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:59.789932    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc" event={"ID":"de922643-08e7-4c2a-9e9a-65a67f902d10","Type":"ContainerStarted","Data":"c747f93c48030481ecedcdfdeb5468ccc63fed5211fc53fbad9f1edab7ae0e1f"}
Oct 31 07:40:59.830542 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:40:59.830483    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc" podStartSLOduration=1.830469337 podStartE2EDuration="1.830469337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:40:59.829085363 +0000 UTC m=+723.968245950" watchObservedRunningTime="2026-10-31 07:40:59.830469337 +0000 UTC m=+723.969629924"
Oct 31 07:41:00.809021 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:00.808978    5829 generic.go:356] "Generic (PLEG): container finished" podID="de922643-08e7-4c2a-9e9a-65a67f902d10" containerID="0b1f14cdabb9ff7844b7fb9e434d344dd48aff5d2cd32c46e4919d0aaca847bf" exitCode=0
Oct 31 07:41:00.809559 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:00.809072    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc" event={"ID":"de922643-08e7-4c2a-9e9a-65a67f902d10","Type":"ContainerDied","Data":"0b1f14cdabb9ff7844b7fb9e434d344dd48aff5d2cd32c46e4919d0aaca847bf"}
Oct 31 07:41:01.968149 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:01.968123    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:41:02.085387 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.085276    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-openshift-ca\") pod \"de922643-08e7-4c2a-9e9a-65a67f902d10\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") "
Oct 31 07:41:02.085552 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.085399    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de922643-08e7-4c2a-9e9a-65a67f902d10-kube-api-access-x685j\" (UniqueName: \"kubernetes.io/projected/de922643-08e7-4c2a-9e9a-65a67f902d10-kube-api-access-x685j\") pod \"de922643-08e7-4c2a-9e9a-65a67f902d10\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") "
Oct 31 07:41:02.085552 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.085530    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-xks-ca\") pod \"de922643-08e7-4c2a-9e9a-65a67f902d10\" (UID: \"de922643-08e7-4c2a-9e9a-65a67f902d10\") "
Oct 31 07:41:02.085718 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.085687    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-openshift-ca" pod "de922643-08e7-4c2a-9e9a-65a67f902d10" (UID: "de922643-08e7-4c2a-9e9a-65a67f902d10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:02.085942 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.085923    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:02.087527 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.087505    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-xks-ca" pod "de922643-08e7-4c2a-9e9a-65a67f902d10" (UID: "de922643-08e7-4c2a-9e9a-65a67f902d10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:02.087527 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.087509    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de922643-08e7-4c2a-9e9a-65a67f902d10-kube-api-access-x685j" pod "de922643-08e7-4c2a-9e9a-65a67f902d10" (UID: "de922643-08e7-4c2a-9e9a-65a67f902d10"). InnerVolumeSpecName "kube-api-access-x685j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:02.187757 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.187723    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de922643-08e7-4c2a-9e9a-65a67f902d10-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:02.187757 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.187759    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x685j\" (UniqueName: \"kubernetes.io/projected/de922643-08e7-4c2a-9e9a-65a67f902d10-kube-api-access-x685j\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:02.822715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.822683    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"
Oct 31 07:41:02.822972 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.822685    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc" event={"ID":"de922643-08e7-4c2a-9e9a-65a67f902d10","Type":"ContainerDied","Data":"c747f93c48030481ecedcdfdeb5468ccc63fed5211fc53fbad9f1edab7ae0e1f"}
Oct 31 07:41:02.822972 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:02.822803    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c747f93c48030481ecedcdfdeb5468ccc63fed5211fc53fbad9f1edab7ae0e1f"
Oct 31 07:41:03.029638 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.029602    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"]
Oct 31 07:41:03.030198 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.030184    5829 cpu_manager.go:414] "containerMap: removing container" podUID="de922643-08e7-4c2a-9e9a-65a67f902d10" containerName="revoke-keys"
Oct 31 07:41:03.030249 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.030198    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de922643-08e7-4c2a-9e9a-65a67f902d10" containerName="revoke-keys"
Oct 31 07:41:03.030328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.030305    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="de922643-08e7-4c2a-9e9a-65a67f902d10" containerName="revoke-keys"
Oct 31 07:41:03.033880 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.033862    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.056157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.056134    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"]
Oct 31 07:41:03.065227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.065199    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3cc201-openshift-default-dockercfg-2c2sr\""
Oct 31 07:41:03.097317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.097214    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-certs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.097317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.097294    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965ba733-f75b-4e9b-b486-443bdea104e2-istiod-ca-cert\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.097528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.097407    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.097528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.097445    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-istio-token\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.097528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.097464    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-credential-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.097528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.097489    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-data\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.097675 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.097634    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965ba733-f75b-4e9b-b486-443bdea104e2-istio-podinfo\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.097710 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.097690    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-envoy\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.097749 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.097714    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6rs\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-kube-api-access-px6rs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.198430 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198396    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965ba733-f75b-4e9b-b486-443bdea104e2-istiod-ca-cert\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.198639 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198478    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.198639 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198498    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-istio-token\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.198639 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198517    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-credential-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.198639 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198550    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-data\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.198639 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198609    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965ba733-f75b-4e9b-b486-443bdea104e2-istio-podinfo\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.198925 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198764    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-envoy\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.198925 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198818    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6rs\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-kube-api-access-px6rs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.198925 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198902    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-certs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.199083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198945    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-credential-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.199083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.198988    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-data\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.199214 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.199178    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.199272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.199222    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-certs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.199272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.199241    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965ba733-f75b-4e9b-b486-443bdea104e2-istiod-ca-cert\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.200934 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.200908    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-envoy\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.201123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.201104    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965ba733-f75b-4e9b-b486-443bdea104e2-istio-podinfo\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.217989 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.217964    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-istio-token\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.217989 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.217980    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6rs\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-kube-api-access-px6rs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.347721 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.347638    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:03.666393 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.666368    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"]
Oct 31 07:41:03.668209 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:03.668183    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod965ba733_f75b_4e9b_b486_443bdea104e2.slice/crio-9a143de8b8c140e0874d9ea0b4dff550c728e02fa985581cc448823776fc4f9a WatchSource:0}: Error finding container 9a143de8b8c140e0874d9ea0b4dff550c728e02fa985581cc448823776fc4f9a: Status 404 returned error can't find the container with id 9a143de8b8c140e0874d9ea0b4dff550c728e02fa985581cc448823776fc4f9a
Oct 31 07:41:03.670882 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.670851    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:41:03.670964 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.670919    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:41:03.670964 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.670946    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:41:03.831854 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.831820    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" event={"ID":"965ba733-f75b-4e9b-b486-443bdea104e2","Type":"ContainerStarted","Data":"4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d"}
Oct 31 07:41:03.832032 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.831878    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" event={"ID":"965ba733-f75b-4e9b-b486-443bdea104e2","Type":"ContainerStarted","Data":"9a143de8b8c140e0874d9ea0b4dff550c728e02fa985581cc448823776fc4f9a"}
Oct 31 07:41:03.875890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:03.875824    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" podStartSLOduration=0.875804374 podStartE2EDuration="875.804374ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:03.87307312 +0000 UTC m=+728.012233712" watchObservedRunningTime="2026-10-31 07:41:03.875804374 +0000 UTC m=+728.014964966"
Oct 31 07:41:04.348337 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:04.348274    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:04.350174 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:04.350148    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 07:41:04.350324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:04.350193    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 07:41:05.348965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:05.348925    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 07:41:05.349426 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:05.348981    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 07:41:06.347940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:06.347909    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 07:41:06.348144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:06.347962    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 07:41:07.347937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.347898    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 07:41:07.348373 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.347965    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 07:41:07.637151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.637119    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"]
Oct 31 07:41:07.641656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.641630    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.660720 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.660686    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"]
Oct 31 07:41:07.669996 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.669940    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:41:07.669996 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.669954    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:41:07.670216 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.670022    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:41:07.670216 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.670041    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:41:07.762333 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.762235    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d24c28d0-88b2b\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.762492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.762374    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d24c28d0-88b2b\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.762492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.762474    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfw2s\" (UniqueName: \"kubernetes.io/projected/13831a00-7448-4583-b6cf-1ed17e2ddaec-kube-api-access-tfw2s\") pod \"maas-api-revoke-sub-d24c28d0-88b2b\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.863472 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.863438    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfw2s\" (UniqueName: \"kubernetes.io/projected/13831a00-7448-4583-b6cf-1ed17e2ddaec-kube-api-access-tfw2s\") pod \"maas-api-revoke-sub-d24c28d0-88b2b\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.863640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.863584    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d24c28d0-88b2b\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.863732 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.863645    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d24c28d0-88b2b\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.864391 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.864371    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d24c28d0-88b2b\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.865979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.865962    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d24c28d0-88b2b\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.883247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.883222    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfw2s\" (UniqueName: \"kubernetes.io/projected/13831a00-7448-4583-b6cf-1ed17e2ddaec-kube-api-access-tfw2s\") pod \"maas-api-revoke-sub-d24c28d0-88b2b\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:07.957332 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:07.957231    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:08.281442 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.281411    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"]
Oct 31 07:41:08.283796 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:08.283767    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13831a00_7448_4583_b6cf_1ed17e2ddaec.slice/crio-6cde4daceabc3b91ac23f6904c50d048ee92b709e205ada1a4e4ff528c1ae152 WatchSource:0}: Error finding container 6cde4daceabc3b91ac23f6904c50d048ee92b709e205ada1a4e4ff528c1ae152: Status 404 returned error can't find the container with id 6cde4daceabc3b91ac23f6904c50d048ee92b709e205ada1a4e4ff528c1ae152
Oct 31 07:41:08.348980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.348944    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 07:41:08.349393 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.349010    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 07:41:08.752870 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.752837    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"]
Oct 31 07:41:08.756905 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.756886    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:08.776138 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.776108    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"]
Oct 31 07:41:08.864709 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.864621    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b" event={"ID":"13831a00-7448-4583-b6cf-1ed17e2ddaec","Type":"ContainerStarted","Data":"d0590e486ea3881a1f79bddd7160f398bb4f7c4960dead068fe1ccdca4d697e3"}
Oct 31 07:41:08.864709 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.864678    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b" event={"ID":"13831a00-7448-4583-b6cf-1ed17e2ddaec","Type":"ContainerStarted","Data":"6cde4daceabc3b91ac23f6904c50d048ee92b709e205ada1a4e4ff528c1ae152"}
Oct 31 07:41:08.876354 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.876320    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb4a57ce-zvnb7\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:08.876494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.876432    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrjqj\" (UniqueName: \"kubernetes.io/projected/cdf75c6b-480f-48be-a51b-f07000cd37df-kube-api-access-qrjqj\") pod \"maas-api-revoke-sub-fb4a57ce-zvnb7\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:08.876534 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.876525    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb4a57ce-zvnb7\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:08.918313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.904642    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b" podStartSLOduration=1.904617762 podStartE2EDuration="1.904617762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:08.902599925 +0000 UTC m=+733.041760514" watchObservedRunningTime="2026-10-31 07:41:08.904617762 +0000 UTC m=+733.043778350"
Oct 31 07:41:08.977513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.977475    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb4a57ce-zvnb7\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:08.977724 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.977597    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrjqj\" (UniqueName: \"kubernetes.io/projected/cdf75c6b-480f-48be-a51b-f07000cd37df-kube-api-access-qrjqj\") pod \"maas-api-revoke-sub-fb4a57ce-zvnb7\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:08.977790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.977760    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb4a57ce-zvnb7\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:08.978163 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.978138    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb4a57ce-zvnb7\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:08.980017 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.979997    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb4a57ce-zvnb7\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:08.996815 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:08.996781    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrjqj\" (UniqueName: \"kubernetes.io/projected/cdf75c6b-480f-48be-a51b-f07000cd37df-kube-api-access-qrjqj\") pod \"maas-api-revoke-sub-fb4a57ce-zvnb7\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:09.069734 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:09.069692    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:09.348554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:09.348517    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Oct 31 07:41:09.348729 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:09.348576    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Oct 31 07:41:09.433481 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:09.433450    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"]
Oct 31 07:41:09.478297 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:09.478246    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf75c6b_480f_48be_a51b_f07000cd37df.slice/crio-b474f347663fcb74034db5aba9597b15d8776b32cda376b9e3e6ccdeeec8e34b WatchSource:0}: Error finding container b474f347663fcb74034db5aba9597b15d8776b32cda376b9e3e6ccdeeec8e34b: Status 404 returned error can't find the container with id b474f347663fcb74034db5aba9597b15d8776b32cda376b9e3e6ccdeeec8e34b
Oct 31 07:41:09.873365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:09.873326    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7" event={"ID":"cdf75c6b-480f-48be-a51b-f07000cd37df","Type":"ContainerStarted","Data":"b474f347663fcb74034db5aba9597b15d8776b32cda376b9e3e6ccdeeec8e34b"}
Oct 31 07:41:09.874977 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:09.874949    5829 generic.go:356] "Generic (PLEG): container finished" podID="13831a00-7448-4583-b6cf-1ed17e2ddaec" containerID="d0590e486ea3881a1f79bddd7160f398bb4f7c4960dead068fe1ccdca4d697e3" exitCode=0
Oct 31 07:41:09.875087 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:09.875042    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b" event={"ID":"13831a00-7448-4583-b6cf-1ed17e2ddaec","Type":"ContainerDied","Data":"d0590e486ea3881a1f79bddd7160f398bb4f7c4960dead068fe1ccdca4d697e3"}
Oct 31 07:41:10.469640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:10.467863    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:10.469640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:10.469199    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:10.472117 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:10.472086    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:41:10.883093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:10.882989    5829 generic.go:356] "Generic (PLEG): container finished" podID="cdf75c6b-480f-48be-a51b-f07000cd37df" containerID="b521173c5a38ccb0bd77ac99ed743bbc5ca1b83d4a14345285ae550130a1adcf" exitCode=0
Oct 31 07:41:10.883255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:10.883088    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7" event={"ID":"cdf75c6b-480f-48be-a51b-f07000cd37df","Type":"ContainerDied","Data":"b521173c5a38ccb0bd77ac99ed743bbc5ca1b83d4a14345285ae550130a1adcf"}
Oct 31 07:41:11.043665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.043636    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:11.099843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.099808    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-xks-ca\") pod \"13831a00-7448-4583-b6cf-1ed17e2ddaec\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") "
Oct 31 07:41:11.100033 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.099916    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-openshift-ca\") pod \"13831a00-7448-4583-b6cf-1ed17e2ddaec\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") "
Oct 31 07:41:11.100098 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.100030    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13831a00-7448-4583-b6cf-1ed17e2ddaec-kube-api-access-tfw2s\" (UniqueName: \"kubernetes.io/projected/13831a00-7448-4583-b6cf-1ed17e2ddaec-kube-api-access-tfw2s\") pod \"13831a00-7448-4583-b6cf-1ed17e2ddaec\" (UID: \"13831a00-7448-4583-b6cf-1ed17e2ddaec\") "
Oct 31 07:41:11.100359 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.100328    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-openshift-ca" pod "13831a00-7448-4583-b6cf-1ed17e2ddaec" (UID: "13831a00-7448-4583-b6cf-1ed17e2ddaec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:11.100812 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.100792    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:11.102327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.102302    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13831a00-7448-4583-b6cf-1ed17e2ddaec-kube-api-access-tfw2s" pod "13831a00-7448-4583-b6cf-1ed17e2ddaec" (UID: "13831a00-7448-4583-b6cf-1ed17e2ddaec"). InnerVolumeSpecName "kube-api-access-tfw2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:11.102466 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.102451    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-xks-ca" pod "13831a00-7448-4583-b6cf-1ed17e2ddaec" (UID: "13831a00-7448-4583-b6cf-1ed17e2ddaec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:11.202538 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.202502    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfw2s\" (UniqueName: \"kubernetes.io/projected/13831a00-7448-4583-b6cf-1ed17e2ddaec-kube-api-access-tfw2s\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:11.202538 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.202533    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13831a00-7448-4583-b6cf-1ed17e2ddaec-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:11.890463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.890433    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"
Oct 31 07:41:11.890463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.890447    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b" event={"ID":"13831a00-7448-4583-b6cf-1ed17e2ddaec","Type":"ContainerDied","Data":"6cde4daceabc3b91ac23f6904c50d048ee92b709e205ada1a4e4ff528c1ae152"}
Oct 31 07:41:11.891015 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:11.890533    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cde4daceabc3b91ac23f6904c50d048ee92b709e205ada1a4e4ff528c1ae152"
Oct 31 07:41:12.032410 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.032379    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:12.112709 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.112667    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdf75c6b-480f-48be-a51b-f07000cd37df-kube-api-access-qrjqj\" (UniqueName: \"kubernetes.io/projected/cdf75c6b-480f-48be-a51b-f07000cd37df-kube-api-access-qrjqj\") pod \"cdf75c6b-480f-48be-a51b-f07000cd37df\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") "
Oct 31 07:41:12.112940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.112921    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-openshift-ca\") pod \"cdf75c6b-480f-48be-a51b-f07000cd37df\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") "
Oct 31 07:41:12.113090 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.112998    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-xks-ca\") pod \"cdf75c6b-480f-48be-a51b-f07000cd37df\" (UID: \"cdf75c6b-480f-48be-a51b-f07000cd37df\") "
Oct 31 07:41:12.113354 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.113323    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-openshift-ca" pod "cdf75c6b-480f-48be-a51b-f07000cd37df" (UID: "cdf75c6b-480f-48be-a51b-f07000cd37df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:12.113641 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.113620    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:12.115044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.115024    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-xks-ca" pod "cdf75c6b-480f-48be-a51b-f07000cd37df" (UID: "cdf75c6b-480f-48be-a51b-f07000cd37df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:12.115529 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.115509    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf75c6b-480f-48be-a51b-f07000cd37df-kube-api-access-qrjqj" pod "cdf75c6b-480f-48be-a51b-f07000cd37df" (UID: "cdf75c6b-480f-48be-a51b-f07000cd37df"). InnerVolumeSpecName "kube-api-access-qrjqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:12.215343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.215245    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdf75c6b-480f-48be-a51b-f07000cd37df-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:12.215343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.215275    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrjqj\" (UniqueName: \"kubernetes.io/projected/cdf75c6b-480f-48be-a51b-f07000cd37df-kube-api-access-qrjqj\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:12.898336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.898302    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"
Oct 31 07:41:12.898336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.898320    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7" event={"ID":"cdf75c6b-480f-48be-a51b-f07000cd37df","Type":"ContainerDied","Data":"b474f347663fcb74034db5aba9597b15d8776b32cda376b9e3e6ccdeeec8e34b"}
Oct 31 07:41:12.898781 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:12.898378    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b474f347663fcb74034db5aba9597b15d8776b32cda376b9e3e6ccdeeec8e34b"
Oct 31 07:41:14.350022 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:14.349984    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:41:16.510647 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:41:16.510611    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:41:16.566094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.566056    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"]
Oct 31 07:41:16.566708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.566691    5829 cpu_manager.go:414] "containerMap: removing container" podUID="13831a00-7448-4583-b6cf-1ed17e2ddaec" containerName="revoke-keys"
Oct 31 07:41:16.566767 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.566710    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13831a00-7448-4583-b6cf-1ed17e2ddaec" containerName="revoke-keys"
Oct 31 07:41:16.566767 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.566749    5829 cpu_manager.go:414] "containerMap: removing container" podUID="cdf75c6b-480f-48be-a51b-f07000cd37df" containerName="revoke-keys"
Oct 31 07:41:16.566767 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.566754    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdf75c6b-480f-48be-a51b-f07000cd37df" containerName="revoke-keys"
Oct 31 07:41:16.566853 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.566834    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdf75c6b-480f-48be-a51b-f07000cd37df" containerName="revoke-keys"
Oct 31 07:41:16.566853 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.566845    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="13831a00-7448-4583-b6cf-1ed17e2ddaec" containerName="revoke-keys"
Oct 31 07:41:16.574817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.574798    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.589328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.589268    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"]
Oct 31 07:41:16.598850 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.598820    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:41:16.599064 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.598819    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:41:16.602937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.602902    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:41:16.603047 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.603005    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:41:16.670033 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.670000    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99c7c66-hbpp8\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.670232 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.670056    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99c7c66-hbpp8\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.670232 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.670176    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7nm\" (UniqueName: \"kubernetes.io/projected/4434a386-3d84-41f3-929f-ec2960f03ba6-kube-api-access-mg7nm\") pod \"maas-api-revoke-sub-a99c7c66-hbpp8\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.771483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.771400    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99c7c66-hbpp8\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.771483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.771461    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99c7c66-hbpp8\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.771668 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.771502    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7nm\" (UniqueName: \"kubernetes.io/projected/4434a386-3d84-41f3-929f-ec2960f03ba6-kube-api-access-mg7nm\") pod \"maas-api-revoke-sub-a99c7c66-hbpp8\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.772146 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.772119    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a99c7c66-hbpp8\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.773736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.773711    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a99c7c66-hbpp8\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.790903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.790876    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7nm\" (UniqueName: \"kubernetes.io/projected/4434a386-3d84-41f3-929f-ec2960f03ba6-kube-api-access-mg7nm\") pod \"maas-api-revoke-sub-a99c7c66-hbpp8\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:16.889726 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:16.889692    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:17.194862 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:17.194839    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"]
Oct 31 07:41:17.197417 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:17.197389    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4434a386_3d84_41f3_929f_ec2960f03ba6.slice/crio-fe612d36352ea943aa57d02b3d26aae9c9cc656b171feed5235910b83a470004 WatchSource:0}: Error finding container fe612d36352ea943aa57d02b3d26aae9c9cc656b171feed5235910b83a470004: Status 404 returned error can't find the container with id fe612d36352ea943aa57d02b3d26aae9c9cc656b171feed5235910b83a470004
Oct 31 07:41:17.940504 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:17.940469    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8" event={"ID":"4434a386-3d84-41f3-929f-ec2960f03ba6","Type":"ContainerStarted","Data":"cd0af04ea5bab4a3c81abc90de29209f3b61c0fdea799e5a5cd1ac07cf5dda4a"}
Oct 31 07:41:17.940904 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:17.940528    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8" event={"ID":"4434a386-3d84-41f3-929f-ec2960f03ba6","Type":"ContainerStarted","Data":"fe612d36352ea943aa57d02b3d26aae9c9cc656b171feed5235910b83a470004"}
Oct 31 07:41:17.980334 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:17.980263    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8" podStartSLOduration=1.980243929 podStartE2EDuration="1.980243929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:17.978972604 +0000 UTC m=+742.118133191" watchObservedRunningTime="2026-10-31 07:41:17.980243929 +0000 UTC m=+742.119404519"
Oct 31 07:41:18.645857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.645824    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"]
Oct 31 07:41:18.649992 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.649974    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.670345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.670315    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"]
Oct 31 07:41:18.679818 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.679794    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-648fbe-openshift-default-dockercfg-2vh69\""
Oct 31 07:41:18.796243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.796198    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-certs\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.796462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.796359    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-token\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.796462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.796409    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-data\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.796462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.796435    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.796592 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.796571    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-envoy\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.796628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.796620    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istiod-ca-cert\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.796699 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.796684    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-credential-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.796742 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.796716    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-podinfo\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.796781 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.796742    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmn2\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-kube-api-access-7vmn2\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898159 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898052    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istiod-ca-cert\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898159 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898124    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-credential-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898159    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-podinfo\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898185    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmn2\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-kube-api-access-7vmn2\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898251    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-certs\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898350    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-token\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898499    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-data\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898559    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898634    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-credential-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898651    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-envoy\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898866 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898668    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-certs\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898910    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istiod-ca-cert\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.898983 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.898936    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-data\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.899070 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.899037    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.900733 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.900710    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-envoy\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.900733 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.900722    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-podinfo\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.917753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.917719    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-token\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.917905 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.917804    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmn2\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-kube-api-access-7vmn2\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:18.948698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.948646    5829 generic.go:356] "Generic (PLEG): container finished" podID="4434a386-3d84-41f3-929f-ec2960f03ba6" containerID="cd0af04ea5bab4a3c81abc90de29209f3b61c0fdea799e5a5cd1ac07cf5dda4a" exitCode=0
Oct 31 07:41:18.949074 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.948716    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8" event={"ID":"4434a386-3d84-41f3-929f-ec2960f03ba6","Type":"ContainerDied","Data":"cd0af04ea5bab4a3c81abc90de29209f3b61c0fdea799e5a5cd1ac07cf5dda4a"}
Oct 31 07:41:18.964261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:18.964227    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:19.291193 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:19.291166    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"]
Oct 31 07:41:19.293779 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:19.293747    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3575b5_7dbe_4c72_a538_d50802fc8f5a.slice/crio-15b1a6c7cd971f5c7e6b222d57206653121cdd8c84828d39b7b82e0c40473997 WatchSource:0}: Error finding container 15b1a6c7cd971f5c7e6b222d57206653121cdd8c84828d39b7b82e0c40473997: Status 404 returned error can't find the container with id 15b1a6c7cd971f5c7e6b222d57206653121cdd8c84828d39b7b82e0c40473997
Oct 31 07:41:19.296335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:19.296259    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:41:19.296423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:19.296373    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:41:19.296423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:19.296405    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:41:19.956837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:19.956799    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" event={"ID":"4f3575b5-7dbe-4c72-a538-d50802fc8f5a","Type":"ContainerStarted","Data":"8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62"}
Oct 31 07:41:19.957312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:19.956860    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" event={"ID":"4f3575b5-7dbe-4c72-a538-d50802fc8f5a","Type":"ContainerStarted","Data":"15b1a6c7cd971f5c7e6b222d57206653121cdd8c84828d39b7b82e0c40473997"}
Oct 31 07:41:19.964461 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:19.964429    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:20.031306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.030407    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" podStartSLOduration=2.030382945 podStartE2EDuration="2.030382945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:20.010879997 +0000 UTC m=+744.150040612" watchObservedRunningTime="2026-10-31 07:41:20.030382945 +0000 UTC m=+744.169543536"
Oct 31 07:41:20.148868 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.148843    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:20.336292 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.336190    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4434a386-3d84-41f3-929f-ec2960f03ba6-kube-api-access-mg7nm\" (UniqueName: \"kubernetes.io/projected/4434a386-3d84-41f3-929f-ec2960f03ba6-kube-api-access-mg7nm\") pod \"4434a386-3d84-41f3-929f-ec2960f03ba6\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") "
Oct 31 07:41:20.336292 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.336233    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-openshift-ca\") pod \"4434a386-3d84-41f3-929f-ec2960f03ba6\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") "
Oct 31 07:41:20.336528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.336313    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-xks-ca\") pod \"4434a386-3d84-41f3-929f-ec2960f03ba6\" (UID: \"4434a386-3d84-41f3-929f-ec2960f03ba6\") "
Oct 31 07:41:20.336719 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.336691    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-openshift-ca" pod "4434a386-3d84-41f3-929f-ec2960f03ba6" (UID: "4434a386-3d84-41f3-929f-ec2960f03ba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:20.338572 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.338538    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-xks-ca" pod "4434a386-3d84-41f3-929f-ec2960f03ba6" (UID: "4434a386-3d84-41f3-929f-ec2960f03ba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:20.338866 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.338842    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4434a386-3d84-41f3-929f-ec2960f03ba6-kube-api-access-mg7nm" pod "4434a386-3d84-41f3-929f-ec2960f03ba6" (UID: "4434a386-3d84-41f3-929f-ec2960f03ba6"). InnerVolumeSpecName "kube-api-access-mg7nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:20.437754 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.437720    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg7nm\" (UniqueName: \"kubernetes.io/projected/4434a386-3d84-41f3-929f-ec2960f03ba6-kube-api-access-mg7nm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:20.437754 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.437750    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:20.437754 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.437761    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4434a386-3d84-41f3-929f-ec2960f03ba6-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:20.965745 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.965709    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"
Oct 31 07:41:20.965745 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.965738    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8" event={"ID":"4434a386-3d84-41f3-929f-ec2960f03ba6","Type":"ContainerDied","Data":"fe612d36352ea943aa57d02b3d26aae9c9cc656b171feed5235910b83a470004"}
Oct 31 07:41:20.966317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.965785    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe612d36352ea943aa57d02b3d26aae9c9cc656b171feed5235910b83a470004"
Oct 31 07:41:20.966317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.966258    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 07:41:20.966437 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:20.966314    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 07:41:21.965085 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:21.965045    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 07:41:21.965299 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:21.965112    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 07:41:22.964782 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:22.964742    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 07:41:22.965190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:22.964805    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 07:41:23.180367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.180329    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"]
Oct 31 07:41:23.180975 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.180960    5829 cpu_manager.go:414] "containerMap: removing container" podUID="4434a386-3d84-41f3-929f-ec2960f03ba6" containerName="revoke-keys"
Oct 31 07:41:23.180975 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.180976    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4434a386-3d84-41f3-929f-ec2960f03ba6" containerName="revoke-keys"
Oct 31 07:41:23.181073 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.181068    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="4434a386-3d84-41f3-929f-ec2960f03ba6" containerName="revoke-keys"
Oct 31 07:41:23.184804 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.184787    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.203261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.203237    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"]
Oct 31 07:41:23.229368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.226906    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:41:23.229368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.227194    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:41:23.229368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.227232    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:41:23.229368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.227469    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:41:23.264326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.264293    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e990ba-pkb7p\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.264521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.264473    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgh7j\" (UniqueName: \"kubernetes.io/projected/c236dbaa-6211-4c66-9057-eeb37250003f-kube-api-access-wgh7j\") pod \"maas-api-revoke-sub-85e990ba-pkb7p\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.264601 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.264578    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e990ba-pkb7p\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.365442 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.365407    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e990ba-pkb7p\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.365655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.365503    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e990ba-pkb7p\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.365655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.365632    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgh7j\" (UniqueName: \"kubernetes.io/projected/c236dbaa-6211-4c66-9057-eeb37250003f-kube-api-access-wgh7j\") pod \"maas-api-revoke-sub-85e990ba-pkb7p\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.366112 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.366086    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e990ba-pkb7p\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.367920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.367902    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e990ba-pkb7p\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.385054 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.385022    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgh7j\" (UniqueName: \"kubernetes.io/projected/c236dbaa-6211-4c66-9057-eeb37250003f-kube-api-access-wgh7j\") pod \"maas-api-revoke-sub-85e990ba-pkb7p\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.498740 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.498656    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:23.830331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.830223    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"]
Oct 31 07:41:23.835134 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:23.835102    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc236dbaa_6211_4c66_9057_eeb37250003f.slice/crio-2ec7b27ae0c2cef3cacbc49357031f264b4524f973ef422e58e12c6a1f4247dc WatchSource:0}: Error finding container 2ec7b27ae0c2cef3cacbc49357031f264b4524f973ef422e58e12c6a1f4247dc: Status 404 returned error can't find the container with id 2ec7b27ae0c2cef3cacbc49357031f264b4524f973ef422e58e12c6a1f4247dc
Oct 31 07:41:23.964910 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.964870    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 07:41:23.965388 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.964945    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 07:41:23.988203 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:23.988169    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p" event={"ID":"c236dbaa-6211-4c66-9057-eeb37250003f","Type":"ContainerStarted","Data":"2ec7b27ae0c2cef3cacbc49357031f264b4524f973ef422e58e12c6a1f4247dc"}
Oct 31 07:41:24.488044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.487953    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"]
Oct 31 07:41:24.493051 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.493033    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.535596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.535566    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"]
Oct 31 07:41:24.547713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.547675    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"]
Oct 31 07:41:24.552984 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.552956    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.571908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.571876    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"]
Oct 31 07:41:24.578702 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.578657    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd02f45f-kltxs\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.579044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.579013    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518aac12-8p8tj\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.579136 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.579116    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518aac12-8p8tj\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.579227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.579210    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd02f45f-kltxs\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.579320 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.579302    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfh2p\" (UniqueName: \"kubernetes.io/projected/53837a9b-2b09-467f-8a73-aaf5b437c068-kube-api-access-hfh2p\") pod \"maas-api-revoke-sub-518aac12-8p8tj\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.579387 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.579339    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xlk\" (UniqueName: \"kubernetes.io/projected/dbee989c-47ec-4629-b7f3-25394499b7f9-kube-api-access-x6xlk\") pod \"maas-api-revoke-sub-dd02f45f-kltxs\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.680363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.680322    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd02f45f-kltxs\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.680520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.680422    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518aac12-8p8tj\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.680520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.680487    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518aac12-8p8tj\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.680632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.680566    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd02f45f-kltxs\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.680632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.680615    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfh2p\" (UniqueName: \"kubernetes.io/projected/53837a9b-2b09-467f-8a73-aaf5b437c068-kube-api-access-hfh2p\") pod \"maas-api-revoke-sub-518aac12-8p8tj\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.680703 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.680650    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xlk\" (UniqueName: \"kubernetes.io/projected/dbee989c-47ec-4629-b7f3-25394499b7f9-kube-api-access-x6xlk\") pod \"maas-api-revoke-sub-dd02f45f-kltxs\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.681386 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.681358    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518aac12-8p8tj\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.681549 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.681358    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd02f45f-kltxs\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.682803 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.682782    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518aac12-8p8tj\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.682803 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.682795    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd02f45f-kltxs\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.699955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.699920    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xlk\" (UniqueName: \"kubernetes.io/projected/dbee989c-47ec-4629-b7f3-25394499b7f9-kube-api-access-x6xlk\") pod \"maas-api-revoke-sub-dd02f45f-kltxs\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.699955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.699948    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfh2p\" (UniqueName: \"kubernetes.io/projected/53837a9b-2b09-467f-8a73-aaf5b437c068-kube-api-access-hfh2p\") pod \"maas-api-revoke-sub-518aac12-8p8tj\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.831605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.831507    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:24.869469 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.869440    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:24.964804 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.964769    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 31 07:41:24.964958 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.964824    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 31 07:41:24.997994 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.997961    5829 generic.go:356] "Generic (PLEG): container finished" podID="c236dbaa-6211-4c66-9057-eeb37250003f" containerID="cc2919452a057a5e3eb447b345bbf524d94030e748e74a5c48108f6b845de8e4" exitCode=0
Oct 31 07:41:24.998179 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:24.998046    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p" event={"ID":"c236dbaa-6211-4c66-9057-eeb37250003f","Type":"ContainerDied","Data":"cc2919452a057a5e3eb447b345bbf524d94030e748e74a5c48108f6b845de8e4"}
Oct 31 07:41:25.150156 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:25.150124    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"]
Oct 31 07:41:25.152131 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:25.152104    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbee989c_47ec_4629_b7f3_25394499b7f9.slice/crio-3a64762d85bb5f539b1c7d419aa30f23a3f5bdf472b2d4dcb01645313da8f463 WatchSource:0}: Error finding container 3a64762d85bb5f539b1c7d419aa30f23a3f5bdf472b2d4dcb01645313da8f463: Status 404 returned error can't find the container with id 3a64762d85bb5f539b1c7d419aa30f23a3f5bdf472b2d4dcb01645313da8f463
Oct 31 07:41:25.222043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:25.222016    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"]
Oct 31 07:41:25.306527 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:25.306486    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53837a9b_2b09_467f_8a73_aaf5b437c068.slice/crio-06d10c992a707620cb8ad20a5733d0f5bcdd4a036c553f4b13470e20231d5367 WatchSource:0}: Error finding container 06d10c992a707620cb8ad20a5733d0f5bcdd4a036c553f4b13470e20231d5367: Status 404 returned error can't find the container with id 06d10c992a707620cb8ad20a5733d0f5bcdd4a036c553f4b13470e20231d5367
Oct 31 07:41:25.998470 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:25.998439    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:25.999270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:25.999131    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:25.999630 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:25.999611    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:41:26.008000 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.007969    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj" event={"ID":"53837a9b-2b09-467f-8a73-aaf5b437c068","Type":"ContainerStarted","Data":"6ca517689cc6c558a83a9d722116d3b37ea210395d3ab063d0a509ef3ef6aec5"}
Oct 31 07:41:26.008164 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.008026    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj" event={"ID":"53837a9b-2b09-467f-8a73-aaf5b437c068","Type":"ContainerStarted","Data":"06d10c992a707620cb8ad20a5733d0f5bcdd4a036c553f4b13470e20231d5367"}
Oct 31 07:41:26.010002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.009953    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs" event={"ID":"dbee989c-47ec-4629-b7f3-25394499b7f9","Type":"ContainerStarted","Data":"eda8c43cc60836a744e5e9c8fabb4d51f806ce46d881750323efb2ff178ce89b"}
Oct 31 07:41:26.010108 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.010003    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs" event={"ID":"dbee989c-47ec-4629-b7f3-25394499b7f9","Type":"ContainerStarted","Data":"3a64762d85bb5f539b1c7d419aa30f23a3f5bdf472b2d4dcb01645313da8f463"}
Oct 31 07:41:26.083477 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.083413    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj" podStartSLOduration=2.083393259 podStartE2EDuration="2.083393259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:26.079233859 +0000 UTC m=+750.218394448" watchObservedRunningTime="2026-10-31 07:41:26.083393259 +0000 UTC m=+750.222553854"
Oct 31 07:41:26.165503 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.165441    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs" podStartSLOduration=2.165420956 podStartE2EDuration="2.165420956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:26.163203149 +0000 UTC m=+750.302363738" watchObservedRunningTime="2026-10-31 07:41:26.165420956 +0000 UTC m=+750.304581568"
Oct 31 07:41:26.205651 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.205626    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:26.312651 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.312620    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c236dbaa-6211-4c66-9057-eeb37250003f-kube-api-access-wgh7j\" (UniqueName: \"kubernetes.io/projected/c236dbaa-6211-4c66-9057-eeb37250003f-kube-api-access-wgh7j\") pod \"c236dbaa-6211-4c66-9057-eeb37250003f\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") "
Oct 31 07:41:26.312839 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.312752    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-xks-ca\") pod \"c236dbaa-6211-4c66-9057-eeb37250003f\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") "
Oct 31 07:41:26.312939 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.312922    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-openshift-ca\") pod \"c236dbaa-6211-4c66-9057-eeb37250003f\" (UID: \"c236dbaa-6211-4c66-9057-eeb37250003f\") "
Oct 31 07:41:26.313336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.313308    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-openshift-ca" pod "c236dbaa-6211-4c66-9057-eeb37250003f" (UID: "c236dbaa-6211-4c66-9057-eeb37250003f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:26.313471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.313423    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:26.314854 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.314828    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-xks-ca" pod "c236dbaa-6211-4c66-9057-eeb37250003f" (UID: "c236dbaa-6211-4c66-9057-eeb37250003f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:26.314854 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.314843    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c236dbaa-6211-4c66-9057-eeb37250003f-kube-api-access-wgh7j" pod "c236dbaa-6211-4c66-9057-eeb37250003f" (UID: "c236dbaa-6211-4c66-9057-eeb37250003f"). InnerVolumeSpecName "kube-api-access-wgh7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:26.414551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.414509    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgh7j\" (UniqueName: \"kubernetes.io/projected/c236dbaa-6211-4c66-9057-eeb37250003f-kube-api-access-wgh7j\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:26.414551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.414545    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c236dbaa-6211-4c66-9057-eeb37250003f-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:26.902790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.902758    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"]
Oct 31 07:41:26.903373 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.903358    5829 cpu_manager.go:414] "containerMap: removing container" podUID="c236dbaa-6211-4c66-9057-eeb37250003f" containerName="revoke-keys"
Oct 31 07:41:26.903441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.903374    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c236dbaa-6211-4c66-9057-eeb37250003f" containerName="revoke-keys"
Oct 31 07:41:26.903493 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.903484    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="c236dbaa-6211-4c66-9057-eeb37250003f" containerName="revoke-keys"
Oct 31 07:41:26.907052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.907037    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:26.926270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:26.926231    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"]
Oct 31 07:41:27.018473 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.018437    5829 generic.go:356] "Generic (PLEG): container finished" podID="53837a9b-2b09-467f-8a73-aaf5b437c068" containerID="6ca517689cc6c558a83a9d722116d3b37ea210395d3ab063d0a509ef3ef6aec5" exitCode=0
Oct 31 07:41:27.018893 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.018534    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj" event={"ID":"53837a9b-2b09-467f-8a73-aaf5b437c068","Type":"ContainerDied","Data":"6ca517689cc6c558a83a9d722116d3b37ea210395d3ab063d0a509ef3ef6aec5"}
Oct 31 07:41:27.020237 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.020209    5829 generic.go:356] "Generic (PLEG): container finished" podID="dbee989c-47ec-4629-b7f3-25394499b7f9" containerID="eda8c43cc60836a744e5e9c8fabb4d51f806ce46d881750323efb2ff178ce89b" exitCode=0
Oct 31 07:41:27.020388 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.020305    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs" event={"ID":"dbee989c-47ec-4629-b7f3-25394499b7f9","Type":"ContainerDied","Data":"eda8c43cc60836a744e5e9c8fabb4d51f806ce46d881750323efb2ff178ce89b"}
Oct 31 07:41:27.021126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.021098    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc8h9\" (UniqueName: \"kubernetes.io/projected/4e81a907-0cd8-4627-9caf-f8c126649d06-kube-api-access-pc8h9\") pod \"maas-api-revoke-sub-98b9bfe2-qq66k\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.021256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.021166    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b9bfe2-qq66k\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.021420 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.021402    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b9bfe2-qq66k\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.021665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.021651    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"
Oct 31 07:41:27.021771 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.021696    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p" event={"ID":"c236dbaa-6211-4c66-9057-eeb37250003f","Type":"ContainerDied","Data":"2ec7b27ae0c2cef3cacbc49357031f264b4524f973ef422e58e12c6a1f4247dc"}
Oct 31 07:41:27.021771 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.021733    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ec7b27ae0c2cef3cacbc49357031f264b4524f973ef422e58e12c6a1f4247dc"
Oct 31 07:41:27.123192 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.123154    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b9bfe2-qq66k\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.123507 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.123479    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b9bfe2-qq66k\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.123755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.123725    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8h9\" (UniqueName: \"kubernetes.io/projected/4e81a907-0cd8-4627-9caf-f8c126649d06-kube-api-access-pc8h9\") pod \"maas-api-revoke-sub-98b9bfe2-qq66k\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.124258 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.124234    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b9bfe2-qq66k\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.125664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.125642    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b9bfe2-qq66k\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.143262 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.143235    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc8h9\" (UniqueName: \"kubernetes.io/projected/4e81a907-0cd8-4627-9caf-f8c126649d06-kube-api-access-pc8h9\") pod \"maas-api-revoke-sub-98b9bfe2-qq66k\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.220199 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.220113    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:27.357023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.356988    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"]
Oct 31 07:41:27.362331 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.362304    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.379488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.379456    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"]
Oct 31 07:41:27.427893 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.427859    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ab5bda7-cw5g7\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.428082 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.427924    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ab5bda7-cw5g7\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.428082 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.428044    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gklsz\" (UniqueName: \"kubernetes.io/projected/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-kube-api-access-gklsz\") pod \"maas-api-revoke-sub-6ab5bda7-cw5g7\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.520093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.520065    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"]
Oct 31 07:41:27.522010 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:27.521983    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e81a907_0cd8_4627_9caf_f8c126649d06.slice/crio-158fa15d7d0c5f692d5fee66979b0e9fad94dcb03f4abc16756c4c19f4a6d8fc WatchSource:0}: Error finding container 158fa15d7d0c5f692d5fee66979b0e9fad94dcb03f4abc16756c4c19f4a6d8fc: Status 404 returned error can't find the container with id 158fa15d7d0c5f692d5fee66979b0e9fad94dcb03f4abc16756c4c19f4a6d8fc
Oct 31 07:41:27.529590 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.529564    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ab5bda7-cw5g7\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.529701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.529608    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gklsz\" (UniqueName: \"kubernetes.io/projected/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-kube-api-access-gklsz\") pod \"maas-api-revoke-sub-6ab5bda7-cw5g7\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.529764 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.529729    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ab5bda7-cw5g7\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.530266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.530246    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ab5bda7-cw5g7\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.532268 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.532243    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ab5bda7-cw5g7\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.549063 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.549038    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gklsz\" (UniqueName: \"kubernetes.io/projected/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-kube-api-access-gklsz\") pod \"maas-api-revoke-sub-6ab5bda7-cw5g7\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.676974 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.676930    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:27.983040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:27.983010    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"]
Oct 31 07:41:27.984502 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:27.984478    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c75c9a5_adcb_4c71_bc3d_b91eddec7668.slice/crio-b150e85b7963086ea0b5927ff2aa6467523b1f2cc8d5fce4ecce158fe2e381b1 WatchSource:0}: Error finding container b150e85b7963086ea0b5927ff2aa6467523b1f2cc8d5fce4ecce158fe2e381b1: Status 404 returned error can't find the container with id b150e85b7963086ea0b5927ff2aa6467523b1f2cc8d5fce4ecce158fe2e381b1
Oct 31 07:41:28.029456 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.029422    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k" event={"ID":"4e81a907-0cd8-4627-9caf-f8c126649d06","Type":"ContainerStarted","Data":"07573383f6254be7a50e3c05159392a1c9493a1d213f4edb159762091a286ed8"}
Oct 31 07:41:28.029878 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.029483    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k" event={"ID":"4e81a907-0cd8-4627-9caf-f8c126649d06","Type":"ContainerStarted","Data":"158fa15d7d0c5f692d5fee66979b0e9fad94dcb03f4abc16756c4c19f4a6d8fc"}
Oct 31 07:41:28.030841 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.030816    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7" event={"ID":"9c75c9a5-adcb-4c71-bc3d-b91eddec7668","Type":"ContainerStarted","Data":"b150e85b7963086ea0b5927ff2aa6467523b1f2cc8d5fce4ecce158fe2e381b1"}
Oct 31 07:41:28.071406 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.071271    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k" podStartSLOduration=2.071253756 podStartE2EDuration="2.071253756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:28.067681233 +0000 UTC m=+752.206841820" watchObservedRunningTime="2026-10-31 07:41:28.071253756 +0000 UTC m=+752.210414343"
Oct 31 07:41:28.216487 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.216459    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:28.218928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.218902    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:28.339574 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.339545    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-xks-ca\") pod \"53837a9b-2b09-467f-8a73-aaf5b437c068\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") "
Oct 31 07:41:28.339732 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.339693    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-xks-ca\") pod \"dbee989c-47ec-4629-b7f3-25394499b7f9\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") "
Oct 31 07:41:28.339798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.339746    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-openshift-ca\") pod \"dbee989c-47ec-4629-b7f3-25394499b7f9\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") "
Oct 31 07:41:28.339798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.339788    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbee989c-47ec-4629-b7f3-25394499b7f9-kube-api-access-x6xlk\" (UniqueName: \"kubernetes.io/projected/dbee989c-47ec-4629-b7f3-25394499b7f9-kube-api-access-x6xlk\") pod \"dbee989c-47ec-4629-b7f3-25394499b7f9\" (UID: \"dbee989c-47ec-4629-b7f3-25394499b7f9\") "
Oct 31 07:41:28.339908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.339875    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-openshift-ca\") pod \"53837a9b-2b09-467f-8a73-aaf5b437c068\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") "
Oct 31 07:41:28.339963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.339950    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53837a9b-2b09-467f-8a73-aaf5b437c068-kube-api-access-hfh2p\" (UniqueName: \"kubernetes.io/projected/53837a9b-2b09-467f-8a73-aaf5b437c068-kube-api-access-hfh2p\") pod \"53837a9b-2b09-467f-8a73-aaf5b437c068\" (UID: \"53837a9b-2b09-467f-8a73-aaf5b437c068\") "
Oct 31 07:41:28.340457 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.340113    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-openshift-ca" pod "dbee989c-47ec-4629-b7f3-25394499b7f9" (UID: "dbee989c-47ec-4629-b7f3-25394499b7f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:28.340457 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.340411    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-openshift-ca" pod "53837a9b-2b09-467f-8a73-aaf5b437c068" (UID: "53837a9b-2b09-467f-8a73-aaf5b437c068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:28.340654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.340634    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:28.340711 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.340660    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:28.342026 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.342003    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbee989c-47ec-4629-b7f3-25394499b7f9-kube-api-access-x6xlk" pod "dbee989c-47ec-4629-b7f3-25394499b7f9" (UID: "dbee989c-47ec-4629-b7f3-25394499b7f9"). InnerVolumeSpecName "kube-api-access-x6xlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:28.342240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.342221    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-xks-ca" pod "dbee989c-47ec-4629-b7f3-25394499b7f9" (UID: "dbee989c-47ec-4629-b7f3-25394499b7f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:28.342355 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.342251    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-xks-ca" pod "53837a9b-2b09-467f-8a73-aaf5b437c068" (UID: "53837a9b-2b09-467f-8a73-aaf5b437c068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:28.342411 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.342396    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53837a9b-2b09-467f-8a73-aaf5b437c068-kube-api-access-hfh2p" pod "53837a9b-2b09-467f-8a73-aaf5b437c068" (UID: "53837a9b-2b09-467f-8a73-aaf5b437c068"). InnerVolumeSpecName "kube-api-access-hfh2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:28.442208 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.442173    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6xlk\" (UniqueName: \"kubernetes.io/projected/dbee989c-47ec-4629-b7f3-25394499b7f9-kube-api-access-x6xlk\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:28.442208 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.442209    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfh2p\" (UniqueName: \"kubernetes.io/projected/53837a9b-2b09-467f-8a73-aaf5b437c068-kube-api-access-hfh2p\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:28.442441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.442228    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53837a9b-2b09-467f-8a73-aaf5b437c068-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:28.442441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:28.442242    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbee989c-47ec-4629-b7f3-25394499b7f9-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:29.045490 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.044814    5829 generic.go:356] "Generic (PLEG): container finished" podID="4e81a907-0cd8-4627-9caf-f8c126649d06" containerID="07573383f6254be7a50e3c05159392a1c9493a1d213f4edb159762091a286ed8" exitCode=0
Oct 31 07:41:29.045490 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.044985    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k" event={"ID":"4e81a907-0cd8-4627-9caf-f8c126649d06","Type":"ContainerDied","Data":"07573383f6254be7a50e3c05159392a1c9493a1d213f4edb159762091a286ed8"}
Oct 31 07:41:29.051237 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.050472    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7" event={"ID":"9c75c9a5-adcb-4c71-bc3d-b91eddec7668","Type":"ContainerStarted","Data":"732b1555131a99468d47bcd797f048df8144681724657ddae3174ce6bcbfb29b"}
Oct 31 07:41:29.057322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.055687    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj" event={"ID":"53837a9b-2b09-467f-8a73-aaf5b437c068","Type":"ContainerDied","Data":"06d10c992a707620cb8ad20a5733d0f5bcdd4a036c553f4b13470e20231d5367"}
Oct 31 07:41:29.057322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.055747    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06d10c992a707620cb8ad20a5733d0f5bcdd4a036c553f4b13470e20231d5367"
Oct 31 07:41:29.057322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.055815    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"
Oct 31 07:41:29.096494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.078707    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs" event={"ID":"dbee989c-47ec-4629-b7f3-25394499b7f9","Type":"ContainerDied","Data":"3a64762d85bb5f539b1c7d419aa30f23a3f5bdf472b2d4dcb01645313da8f463"}
Oct 31 07:41:29.096494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.078775    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a64762d85bb5f539b1c7d419aa30f23a3f5bdf472b2d4dcb01645313da8f463"
Oct 31 07:41:29.096494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.078846    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"
Oct 31 07:41:29.205177 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:29.205111    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7" podStartSLOduration=2.205086928 podStartE2EDuration="2.205086928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:29.178019777 +0000 UTC m=+753.317180365" watchObservedRunningTime="2026-10-31 07:41:29.205086928 +0000 UTC m=+753.344247516"
Oct 31 07:41:30.088389 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.088352    5829 generic.go:356] "Generic (PLEG): container finished" podID="9c75c9a5-adcb-4c71-bc3d-b91eddec7668" containerID="732b1555131a99468d47bcd797f048df8144681724657ddae3174ce6bcbfb29b" exitCode=0
Oct 31 07:41:30.088892 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.088441    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7" event={"ID":"9c75c9a5-adcb-4c71-bc3d-b91eddec7668","Type":"ContainerDied","Data":"732b1555131a99468d47bcd797f048df8144681724657ddae3174ce6bcbfb29b"}
Oct 31 07:41:30.326310 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.321531    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:30.375741 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.374931    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-xks-ca\") pod \"4e81a907-0cd8-4627-9caf-f8c126649d06\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") "
Oct 31 07:41:30.375741 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.375069    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e81a907-0cd8-4627-9caf-f8c126649d06-kube-api-access-pc8h9\" (UniqueName: \"kubernetes.io/projected/4e81a907-0cd8-4627-9caf-f8c126649d06-kube-api-access-pc8h9\") pod \"4e81a907-0cd8-4627-9caf-f8c126649d06\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") "
Oct 31 07:41:30.375741 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.375097    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-openshift-ca\") pod \"4e81a907-0cd8-4627-9caf-f8c126649d06\" (UID: \"4e81a907-0cd8-4627-9caf-f8c126649d06\") "
Oct 31 07:41:30.376078 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.376052    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-openshift-ca" pod "4e81a907-0cd8-4627-9caf-f8c126649d06" (UID: "4e81a907-0cd8-4627-9caf-f8c126649d06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:30.378246 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.378214    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-xks-ca" pod "4e81a907-0cd8-4627-9caf-f8c126649d06" (UID: "4e81a907-0cd8-4627-9caf-f8c126649d06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:30.379665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.379638    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e81a907-0cd8-4627-9caf-f8c126649d06-kube-api-access-pc8h9" pod "4e81a907-0cd8-4627-9caf-f8c126649d06" (UID: "4e81a907-0cd8-4627-9caf-f8c126649d06"). InnerVolumeSpecName "kube-api-access-pc8h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:30.477023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.476984    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:30.477023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.477016    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc8h9\" (UniqueName: \"kubernetes.io/projected/4e81a907-0cd8-4627-9caf-f8c126649d06-kube-api-access-pc8h9\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:30.477023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:30.477028    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e81a907-0cd8-4627-9caf-f8c126649d06-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:31.096368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.096333    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"
Oct 31 07:41:31.096368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.096352    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k" event={"ID":"4e81a907-0cd8-4627-9caf-f8c126649d06","Type":"ContainerDied","Data":"158fa15d7d0c5f692d5fee66979b0e9fad94dcb03f4abc16756c4c19f4a6d8fc"}
Oct 31 07:41:31.096807 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.096403    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="158fa15d7d0c5f692d5fee66979b0e9fad94dcb03f4abc16756c4c19f4a6d8fc"
Oct 31 07:41:31.389833 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.389803    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:31.487668 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.487631    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-kube-api-access-gklsz\" (UniqueName: \"kubernetes.io/projected/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-kube-api-access-gklsz\") pod \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") "
Oct 31 07:41:31.487846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.487711    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-xks-ca\") pod \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") "
Oct 31 07:41:31.487846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.487793    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-openshift-ca\") pod \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\" (UID: \"9c75c9a5-adcb-4c71-bc3d-b91eddec7668\") "
Oct 31 07:41:31.488643 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.488612    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-openshift-ca" pod "9c75c9a5-adcb-4c71-bc3d-b91eddec7668" (UID: "9c75c9a5-adcb-4c71-bc3d-b91eddec7668"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:31.490453 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.490395    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-xks-ca" pod "9c75c9a5-adcb-4c71-bc3d-b91eddec7668" (UID: "9c75c9a5-adcb-4c71-bc3d-b91eddec7668"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:31.490585 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.490448    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-kube-api-access-gklsz" pod "9c75c9a5-adcb-4c71-bc3d-b91eddec7668" (UID: "9c75c9a5-adcb-4c71-bc3d-b91eddec7668"). InnerVolumeSpecName "kube-api-access-gklsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:31.589941 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.589904    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gklsz\" (UniqueName: \"kubernetes.io/projected/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-kube-api-access-gklsz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:31.589941 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.589934    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:31.589941 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:31.589945    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c75c9a5-adcb-4c71-bc3d-b91eddec7668-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:32.103977 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.103946    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"
Oct 31 07:41:32.104467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.103977    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7" event={"ID":"9c75c9a5-adcb-4c71-bc3d-b91eddec7668","Type":"ContainerDied","Data":"b150e85b7963086ea0b5927ff2aa6467523b1f2cc8d5fce4ecce158fe2e381b1"}
Oct 31 07:41:32.104467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.104024    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b150e85b7963086ea0b5927ff2aa6467523b1f2cc8d5fce4ecce158fe2e381b1"
Oct 31 07:41:32.489903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.489867    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"]
Oct 31 07:41:32.490460 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490444    5829 cpu_manager.go:414] "containerMap: removing container" podUID="9c75c9a5-adcb-4c71-bc3d-b91eddec7668" containerName="revoke-keys"
Oct 31 07:41:32.490520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490462    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c75c9a5-adcb-4c71-bc3d-b91eddec7668" containerName="revoke-keys"
Oct 31 07:41:32.490520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490477    5829 cpu_manager.go:414] "containerMap: removing container" podUID="53837a9b-2b09-467f-8a73-aaf5b437c068" containerName="revoke-keys"
Oct 31 07:41:32.490520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490485    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53837a9b-2b09-467f-8a73-aaf5b437c068" containerName="revoke-keys"
Oct 31 07:41:32.490520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490501    5829 cpu_manager.go:414] "containerMap: removing container" podUID="dbee989c-47ec-4629-b7f3-25394499b7f9" containerName="revoke-keys"
Oct 31 07:41:32.490520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490507    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbee989c-47ec-4629-b7f3-25394499b7f9" containerName="revoke-keys"
Oct 31 07:41:32.490728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490543    5829 cpu_manager.go:414] "containerMap: removing container" podUID="4e81a907-0cd8-4627-9caf-f8c126649d06" containerName="revoke-keys"
Oct 31 07:41:32.490728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490548    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e81a907-0cd8-4627-9caf-f8c126649d06" containerName="revoke-keys"
Oct 31 07:41:32.490728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490639    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c75c9a5-adcb-4c71-bc3d-b91eddec7668" containerName="revoke-keys"
Oct 31 07:41:32.490728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490648    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="53837a9b-2b09-467f-8a73-aaf5b437c068" containerName="revoke-keys"
Oct 31 07:41:32.490728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490655    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbee989c-47ec-4629-b7f3-25394499b7f9" containerName="revoke-keys"
Oct 31 07:41:32.490728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.490667    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e81a907-0cd8-4627-9caf-f8c126649d06" containerName="revoke-keys"
Oct 31 07:41:32.494419 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.494398    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.510559 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.510521    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:41:32.510706 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.510577    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:41:32.513969 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.513942    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"]
Oct 31 07:41:32.522413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.522389    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:41:32.522521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.522428    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:41:32.603088 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.603054    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8758b4b-ks5zs\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.603267 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.603125    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbf44\" (UniqueName: \"kubernetes.io/projected/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-kube-api-access-kbf44\") pod \"maas-api-revoke-sub-c8758b4b-ks5zs\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.603267 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.603199    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8758b4b-ks5zs\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.703981 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.703937    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbf44\" (UniqueName: \"kubernetes.io/projected/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-kube-api-access-kbf44\") pod \"maas-api-revoke-sub-c8758b4b-ks5zs\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.704191 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.704003    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8758b4b-ks5zs\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.704191 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.704169    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8758b4b-ks5zs\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.704777 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.704750    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8758b4b-ks5zs\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.706330 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.706308    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8758b4b-ks5zs\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.723593 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.723561    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbf44\" (UniqueName: \"kubernetes.io/projected/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-kube-api-access-kbf44\") pod \"maas-api-revoke-sub-c8758b4b-ks5zs\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:32.808100 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:32.808026    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:33.117586 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:33.117560    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"]
Oct 31 07:41:33.119463 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:33.119435    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f4cd1c_7e7d_4ac3_8f5a_bf9b1615bdf4.slice/crio-7423b87b3dfef3493cbdc11b3d76b3d62c876bd48ec3abcdcf613aaebbbc8f6a WatchSource:0}: Error finding container 7423b87b3dfef3493cbdc11b3d76b3d62c876bd48ec3abcdcf613aaebbbc8f6a: Status 404 returned error can't find the container with id 7423b87b3dfef3493cbdc11b3d76b3d62c876bd48ec3abcdcf613aaebbbc8f6a
Oct 31 07:41:34.119084 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:34.119052    5829 generic.go:356] "Generic (PLEG): container finished" podID="52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4" containerID="cbcda8bd8c54c82cba7197b1358f6405d832331ab8d9e053961d4ca196bf793e" exitCode=0
Oct 31 07:41:34.119503 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:34.119089    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs" event={"ID":"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4","Type":"ContainerDied","Data":"cbcda8bd8c54c82cba7197b1358f6405d832331ab8d9e053961d4ca196bf793e"}
Oct 31 07:41:34.119503 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:34.119129    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs" event={"ID":"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4","Type":"ContainerStarted","Data":"7423b87b3dfef3493cbdc11b3d76b3d62c876bd48ec3abcdcf613aaebbbc8f6a"}
Oct 31 07:41:35.268626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.268591    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:35.344971 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.344926    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-openshift-ca\") pod \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") "
Oct 31 07:41:35.345371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.345348    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-kube-api-access-kbf44\" (UniqueName: \"kubernetes.io/projected/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-kube-api-access-kbf44\") pod \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") "
Oct 31 07:41:35.345493 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.345392    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-xks-ca\") pod \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\" (UID: \"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4\") "
Oct 31 07:41:35.345553 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.345497    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-openshift-ca" pod "52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4" (UID: "52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:35.345762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.345742    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:35.348066 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.348035    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-xks-ca" pod "52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4" (UID: "52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:35.348066 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.348055    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-kube-api-access-kbf44" pod "52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4" (UID: "52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4"). InnerVolumeSpecName "kube-api-access-kbf44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:35.446631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.446596    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbf44\" (UniqueName: \"kubernetes.io/projected/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-kube-api-access-kbf44\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:35.446631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:35.446625    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:36.133677 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:36.133634    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs" event={"ID":"52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4","Type":"ContainerDied","Data":"7423b87b3dfef3493cbdc11b3d76b3d62c876bd48ec3abcdcf613aaebbbc8f6a"}
Oct 31 07:41:36.133677 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:36.133665    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"
Oct 31 07:41:36.133893 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:36.133694    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7423b87b3dfef3493cbdc11b3d76b3d62c876bd48ec3abcdcf613aaebbbc8f6a"
Oct 31 07:41:38.211721 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.211683    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"]
Oct 31 07:41:38.212512 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.212490    5829 cpu_manager.go:414] "containerMap: removing container" podUID="52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4" containerName="revoke-keys"
Oct 31 07:41:38.212635 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.212513    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4" containerName="revoke-keys"
Oct 31 07:41:38.212733 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.212718    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4" containerName="revoke-keys"
Oct 31 07:41:38.216574 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.216551    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.238207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.238174    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"]
Oct 31 07:41:38.246338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.246315    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:41:38.246461 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.246409    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:41:38.263790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.263764    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:41:38.263966 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.263893    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:41:38.284247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.284215    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txfzv\" (UniqueName: \"kubernetes.io/projected/93edee68-d792-4393-ace2-e2a716cb6d0a-kube-api-access-txfzv\") pod \"maas-api-revoke-sub-45190c6a-xpssh\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.284413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.284392    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45190c6a-xpssh\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.284457 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.284424    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45190c6a-xpssh\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.385680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.385641    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txfzv\" (UniqueName: \"kubernetes.io/projected/93edee68-d792-4393-ace2-e2a716cb6d0a-kube-api-access-txfzv\") pod \"maas-api-revoke-sub-45190c6a-xpssh\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.385843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.385753    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45190c6a-xpssh\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.385843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.385777    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45190c6a-xpssh\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.386405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.386385    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45190c6a-xpssh\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.390890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.388657    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45190c6a-xpssh\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.405599 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.405571    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txfzv\" (UniqueName: \"kubernetes.io/projected/93edee68-d792-4393-ace2-e2a716cb6d0a-kube-api-access-txfzv\") pod \"maas-api-revoke-sub-45190c6a-xpssh\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.530523 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.530437    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:38.839051 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:38.838869    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"]
Oct 31 07:41:38.842197 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:38.842165    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93edee68_d792_4393_ace2_e2a716cb6d0a.slice/crio-626ae7fc54bf8964d0158e669d2f787276d05738c0094f7431c0a4db2b42368f WatchSource:0}: Error finding container 626ae7fc54bf8964d0158e669d2f787276d05738c0094f7431c0a4db2b42368f: Status 404 returned error can't find the container with id 626ae7fc54bf8964d0158e669d2f787276d05738c0094f7431c0a4db2b42368f
Oct 31 07:41:39.155134 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:39.155091    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh" event={"ID":"93edee68-d792-4393-ace2-e2a716cb6d0a","Type":"ContainerStarted","Data":"f67292100cae2f8a17301fcf20e2158c965dee48dd1fd4f484055d7d7c0ea92f"}
Oct 31 07:41:39.155325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:39.155152    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh" event={"ID":"93edee68-d792-4393-ace2-e2a716cb6d0a","Type":"ContainerStarted","Data":"626ae7fc54bf8964d0158e669d2f787276d05738c0094f7431c0a4db2b42368f"}
Oct 31 07:41:39.199437 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:39.199388    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh" podStartSLOduration=1.199374356 podStartE2EDuration="1.199374356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:39.196149328 +0000 UTC m=+763.335309918" watchObservedRunningTime="2026-10-31 07:41:39.199374356 +0000 UTC m=+763.338534944"
Oct 31 07:41:40.163695 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:40.163656    5829 generic.go:356] "Generic (PLEG): container finished" podID="93edee68-d792-4393-ace2-e2a716cb6d0a" containerID="f67292100cae2f8a17301fcf20e2158c965dee48dd1fd4f484055d7d7c0ea92f" exitCode=0
Oct 31 07:41:40.164099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:40.163731    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh" event={"ID":"93edee68-d792-4393-ace2-e2a716cb6d0a","Type":"ContainerDied","Data":"f67292100cae2f8a17301fcf20e2158c965dee48dd1fd4f484055d7d7c0ea92f"}
Oct 31 07:41:41.323915 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.323887    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:41.437360 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.436510    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93edee68-d792-4393-ace2-e2a716cb6d0a-kube-api-access-txfzv\" (UniqueName: \"kubernetes.io/projected/93edee68-d792-4393-ace2-e2a716cb6d0a-kube-api-access-txfzv\") pod \"93edee68-d792-4393-ace2-e2a716cb6d0a\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") "
Oct 31 07:41:41.437360 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.436819    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-openshift-ca\") pod \"93edee68-d792-4393-ace2-e2a716cb6d0a\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") "
Oct 31 07:41:41.437360 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.436858    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-xks-ca\") pod \"93edee68-d792-4393-ace2-e2a716cb6d0a\" (UID: \"93edee68-d792-4393-ace2-e2a716cb6d0a\") "
Oct 31 07:41:41.454348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.448882    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93edee68-d792-4393-ace2-e2a716cb6d0a-kube-api-access-txfzv" pod "93edee68-d792-4393-ace2-e2a716cb6d0a" (UID: "93edee68-d792-4393-ace2-e2a716cb6d0a"). InnerVolumeSpecName "kube-api-access-txfzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:41.454348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.449260    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-openshift-ca" pod "93edee68-d792-4393-ace2-e2a716cb6d0a" (UID: "93edee68-d792-4393-ace2-e2a716cb6d0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:41.454348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.452383    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:41.454348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.452412    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txfzv\" (UniqueName: \"kubernetes.io/projected/93edee68-d792-4393-ace2-e2a716cb6d0a-kube-api-access-txfzv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:41.484304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.473415    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-xks-ca" pod "93edee68-d792-4393-ace2-e2a716cb6d0a" (UID: "93edee68-d792-4393-ace2-e2a716cb6d0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:41.553771 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.553738    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93edee68-d792-4393-ace2-e2a716cb6d0a-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:41.654929 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.654889    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"]
Oct 31 07:41:41.655557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.655536    5829 cpu_manager.go:414] "containerMap: removing container" podUID="93edee68-d792-4393-ace2-e2a716cb6d0a" containerName="revoke-keys"
Oct 31 07:41:41.655557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.655555    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93edee68-d792-4393-ace2-e2a716cb6d0a" containerName="revoke-keys"
Oct 31 07:41:41.655715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.655676    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="93edee68-d792-4393-ace2-e2a716cb6d0a" containerName="revoke-keys"
Oct 31 07:41:41.659739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.659717    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.689843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.689765    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"]
Oct 31 07:41:41.756580 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.756537    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.756762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.756612    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.756762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.756710    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49dc456-e746-4d51-94ab-124f8abeeb49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.756762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.756753    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.756874 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.756852    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.756913 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.756900    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.756957 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.756945    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzhvp\" (UniqueName: \"kubernetes.io/projected/a49dc456-e746-4d51-94ab-124f8abeeb49-kube-api-access-nzhvp\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.757024 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.757013    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.757059 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.757037    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858056    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858100    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858138    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858186    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858228    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49dc456-e746-4d51-94ab-124f8abeeb49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858262    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858367    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858642    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-workload-certs\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858651    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858699    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-data\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858708    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzhvp\" (UniqueName: \"kubernetes.io/projected/a49dc456-e746-4d51-94ab-124f8abeeb49-kube-api-access-nzhvp\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858774    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-credential-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.858948 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.858900    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-workload-socket\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.859101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.859079    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a49dc456-e746-4d51-94ab-124f8abeeb49-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.860735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.860713    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-envoy\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.860973 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.860949    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-podinfo\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.878576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.878550    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a49dc456-e746-4d51-94ab-124f8abeeb49-istio-token\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.878703 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.878633    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzhvp\" (UniqueName: \"kubernetes.io/projected/a49dc456-e746-4d51-94ab-124f8abeeb49-kube-api-access-nzhvp\") pod \"maas-default-gateway-openshift-default-645cf464d7-m8njr\" (UID: \"a49dc456-e746-4d51-94ab-124f8abeeb49\") " pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:41.976023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:41.975941    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:42.179631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:42.179602    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"
Oct 31 07:41:42.179818 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:42.179630    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh" event={"ID":"93edee68-d792-4393-ace2-e2a716cb6d0a","Type":"ContainerDied","Data":"626ae7fc54bf8964d0158e669d2f787276d05738c0094f7431c0a4db2b42368f"}
Oct 31 07:41:42.179818 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:42.179684    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="626ae7fc54bf8964d0158e669d2f787276d05738c0094f7431c0a4db2b42368f"
Oct 31 07:41:42.302173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:42.302131    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"]
Oct 31 07:41:42.306293 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:42.306227    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:41:42.306429 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:42.306398    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:41:42.306494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:42.306481    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:41:42.879681 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:42.879638    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:41:43.190838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:43.190799    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr" event={"ID":"a49dc456-e746-4d51-94ab-124f8abeeb49","Type":"ContainerStarted","Data":"51c59feba581328aa6880cbd5515486a6248b61135b80414cd4c1d15785a9683"}
Oct 31 07:41:43.191028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:43.190863    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr" event={"ID":"a49dc456-e746-4d51-94ab-124f8abeeb49","Type":"ContainerStarted","Data":"fc90b65f6e17c5153bcdf1e8783f37889a3ca46b4d6894106389f16342f79c99"}
Oct 31 07:41:43.281108 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:43.281047    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr" podStartSLOduration=2.281030844 podStartE2EDuration="2.281030844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:43.265129704 +0000 UTC m=+767.404290288" watchObservedRunningTime="2026-10-31 07:41:43.281030844 +0000 UTC m=+767.420191469"
Oct 31 07:41:43.976994 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:43.976961    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:43.978104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:43.978071    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-m8njr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 07:41:43.978210 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:43.978141    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr" podUID="a49dc456-e746-4d51-94ab-124f8abeeb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 07:41:44.976611 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:44.976577    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-m8njr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 07:41:44.976765 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:44.976644    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr" podUID="a49dc456-e746-4d51-94ab-124f8abeeb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 07:41:45.976555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:45.976519    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-m8njr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 07:41:45.976948 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:45.976568    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr" podUID="a49dc456-e746-4d51-94ab-124f8abeeb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 07:41:46.537844 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:41:46.537805    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:41:46.976648 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:46.976607    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-m8njr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 07:41:46.977102 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:46.976678    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr" podUID="a49dc456-e746-4d51-94ab-124f8abeeb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 07:41:47.233393 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.233303    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"]
Oct 31 07:41:47.238068 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.238049    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.253830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.253788    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccxmk\" (UniqueName: \"kubernetes.io/projected/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-kube-api-access-ccxmk\") pod \"maas-api-revoke-sub-febff933-2q6pn\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.254009 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.253988    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-febff933-2q6pn\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.254241 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.254190    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-febff933-2q6pn\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.256314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.256267    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"]
Oct 31 07:41:47.265497 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.265461    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:41:47.265497 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.265472    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:41:47.265669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.265559    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:41:47.266440 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.266413    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:41:47.355722 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.355677    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-febff933-2q6pn\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.355904 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.355743    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccxmk\" (UniqueName: \"kubernetes.io/projected/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-kube-api-access-ccxmk\") pod \"maas-api-revoke-sub-febff933-2q6pn\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.355904 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.355874    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-febff933-2q6pn\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.356722 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.356698    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-febff933-2q6pn\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.358128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.358101    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-febff933-2q6pn\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.375462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.375430    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccxmk\" (UniqueName: \"kubernetes.io/projected/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-kube-api-access-ccxmk\") pod \"maas-api-revoke-sub-febff933-2q6pn\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.552818 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.552721    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:47.877173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.877112    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"]
Oct 31 07:41:47.879825 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:47.879796    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7ae355_5f9f_42d8_962e_b398dd9a16cb.slice/crio-50cb31fe3f5708782ae143ce87fe026edd56525e97437ca9a1f9e3ff71d17d64 WatchSource:0}: Error finding container 50cb31fe3f5708782ae143ce87fe026edd56525e97437ca9a1f9e3ff71d17d64: Status 404 returned error can't find the container with id 50cb31fe3f5708782ae143ce87fe026edd56525e97437ca9a1f9e3ff71d17d64
Oct 31 07:41:47.976773 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.976739    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-m8njr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 07:41:47.977162 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:47.976787    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr" podUID="a49dc456-e746-4d51-94ab-124f8abeeb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 07:41:48.255426 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:48.255390    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn" event={"ID":"9a7ae355-5f9f-42d8-962e-b398dd9a16cb","Type":"ContainerStarted","Data":"50cb31fe3f5708782ae143ce87fe026edd56525e97437ca9a1f9e3ff71d17d64"}
Oct 31 07:41:48.976501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:48.976462    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-m8njr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 07:41:48.976684 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:48.976517    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr" podUID="a49dc456-e746-4d51-94ab-124f8abeeb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 07:41:49.267763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:49.267674    5829 generic.go:356] "Generic (PLEG): container finished" podID="9a7ae355-5f9f-42d8-962e-b398dd9a16cb" containerID="881b9f5a003a3044fcce1503b214369176ccd76cf27d229b83a0418812047cbc" exitCode=0
Oct 31 07:41:49.268131 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:49.267761    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn" event={"ID":"9a7ae355-5f9f-42d8-962e-b398dd9a16cb","Type":"ContainerDied","Data":"881b9f5a003a3044fcce1503b214369176ccd76cf27d229b83a0418812047cbc"}
Oct 31 07:41:50.453907 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.453869    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:50.492501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.492456    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-openshift-ca\") pod \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") "
Oct 31 07:41:50.492694 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.492507    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-xks-ca\") pod \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") "
Oct 31 07:41:50.492694 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.492616    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-kube-api-access-ccxmk\" (UniqueName: \"kubernetes.io/projected/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-kube-api-access-ccxmk\") pod \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\" (UID: \"9a7ae355-5f9f-42d8-962e-b398dd9a16cb\") "
Oct 31 07:41:50.493132 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.492798    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-openshift-ca" pod "9a7ae355-5f9f-42d8-962e-b398dd9a16cb" (UID: "9a7ae355-5f9f-42d8-962e-b398dd9a16cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:50.493982 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.493950    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:50.495656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.495601    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-kube-api-access-ccxmk" pod "9a7ae355-5f9f-42d8-962e-b398dd9a16cb" (UID: "9a7ae355-5f9f-42d8-962e-b398dd9a16cb"). InnerVolumeSpecName "kube-api-access-ccxmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:50.496010 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.495983    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-xks-ca" pod "9a7ae355-5f9f-42d8-962e-b398dd9a16cb" (UID: "9a7ae355-5f9f-42d8-962e-b398dd9a16cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:50.596368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.595457    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:50.596368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.595512    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccxmk\" (UniqueName: \"kubernetes.io/projected/9a7ae355-5f9f-42d8-962e-b398dd9a16cb-kube-api-access-ccxmk\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:50.651661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.651617    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"]
Oct 31 07:41:50.652276 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.652243    5829 cpu_manager.go:414] "containerMap: removing container" podUID="9a7ae355-5f9f-42d8-962e-b398dd9a16cb" containerName="revoke-keys"
Oct 31 07:41:50.652276 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.652273    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7ae355-5f9f-42d8-962e-b398dd9a16cb" containerName="revoke-keys"
Oct 31 07:41:50.652491 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.652380    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7ae355-5f9f-42d8-962e-b398dd9a16cb" containerName="revoke-keys"
Oct 31 07:41:50.656078 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.656053    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.676812 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.676778    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"]
Oct 31 07:41:50.696368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.696328    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xb7c\" (UniqueName: \"kubernetes.io/projected/97aadeef-b4a9-4159-ae79-076e95f95cb4-kube-api-access-4xb7c\") pod \"maas-api-revoke-sub-8f468be1-jhgkc\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.696547 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.696400    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f468be1-jhgkc\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.696547 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.696463    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f468be1-jhgkc\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.797611 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.797576    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xb7c\" (UniqueName: \"kubernetes.io/projected/97aadeef-b4a9-4159-ae79-076e95f95cb4-kube-api-access-4xb7c\") pod \"maas-api-revoke-sub-8f468be1-jhgkc\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.797812 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.797623    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f468be1-jhgkc\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.797812 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.797667    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f468be1-jhgkc\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.798398 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.798371    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f468be1-jhgkc\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.800109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.800089    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f468be1-jhgkc\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.802960 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.802934    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:50.803336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.803316    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:50.804231 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.804213    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-m8njr"
Oct 31 07:41:50.817100 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.817077    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xb7c\" (UniqueName: \"kubernetes.io/projected/97aadeef-b4a9-4159-ae79-076e95f95cb4-kube-api-access-4xb7c\") pod \"maas-api-revoke-sub-8f468be1-jhgkc\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:50.979402 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:50.979365    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:51.282959 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:51.282919    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"]
Oct 31 07:41:51.284146 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:51.284119    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"
Oct 31 07:41:51.284146 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:51.284131    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn" event={"ID":"9a7ae355-5f9f-42d8-962e-b398dd9a16cb","Type":"ContainerDied","Data":"50cb31fe3f5708782ae143ce87fe026edd56525e97437ca9a1f9e3ff71d17d64"}
Oct 31 07:41:51.284338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:51.284180    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50cb31fe3f5708782ae143ce87fe026edd56525e97437ca9a1f9e3ff71d17d64"
Oct 31 07:41:51.284338 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:51.284262    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aadeef_b4a9_4159_ae79_076e95f95cb4.slice/crio-66a4586817d53e15d94a44a195b55cbab79974f34cfa93527449cd6588ad49ff WatchSource:0}: Error finding container 66a4586817d53e15d94a44a195b55cbab79974f34cfa93527449cd6588ad49ff: Status 404 returned error can't find the container with id 66a4586817d53e15d94a44a195b55cbab79974f34cfa93527449cd6588ad49ff
Oct 31 07:41:52.293029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:52.292987    5829 generic.go:356] "Generic (PLEG): container finished" podID="97aadeef-b4a9-4159-ae79-076e95f95cb4" containerID="1b8e28897df68a546d0ce9c101e2b50cd08915d16296173cf36182a357b39ed5" exitCode=0
Oct 31 07:41:52.293029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:52.293020    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc" event={"ID":"97aadeef-b4a9-4159-ae79-076e95f95cb4","Type":"ContainerDied","Data":"1b8e28897df68a546d0ce9c101e2b50cd08915d16296173cf36182a357b39ed5"}
Oct 31 07:41:52.293515 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:52.293079    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc" event={"ID":"97aadeef-b4a9-4159-ae79-076e95f95cb4","Type":"ContainerStarted","Data":"66a4586817d53e15d94a44a195b55cbab79974f34cfa93527449cd6588ad49ff"}
Oct 31 07:41:53.452431 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.452405    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:53.532564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.532526    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-xks-ca\") pod \"97aadeef-b4a9-4159-ae79-076e95f95cb4\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") "
Oct 31 07:41:53.532725 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.532629    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-openshift-ca\") pod \"97aadeef-b4a9-4159-ae79-076e95f95cb4\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") "
Oct 31 07:41:53.532789 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.532766    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97aadeef-b4a9-4159-ae79-076e95f95cb4-kube-api-access-4xb7c\" (UniqueName: \"kubernetes.io/projected/97aadeef-b4a9-4159-ae79-076e95f95cb4-kube-api-access-4xb7c\") pod \"97aadeef-b4a9-4159-ae79-076e95f95cb4\" (UID: \"97aadeef-b4a9-4159-ae79-076e95f95cb4\") "
Oct 31 07:41:53.532966 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.532942    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-openshift-ca" pod "97aadeef-b4a9-4159-ae79-076e95f95cb4" (UID: "97aadeef-b4a9-4159-ae79-076e95f95cb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:41:53.533410 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.533390    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:53.534903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.534878    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-xks-ca" pod "97aadeef-b4a9-4159-ae79-076e95f95cb4" (UID: "97aadeef-b4a9-4159-ae79-076e95f95cb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:41:53.535028 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.535003    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97aadeef-b4a9-4159-ae79-076e95f95cb4-kube-api-access-4xb7c" pod "97aadeef-b4a9-4159-ae79-076e95f95cb4" (UID: "97aadeef-b4a9-4159-ae79-076e95f95cb4"). InnerVolumeSpecName "kube-api-access-4xb7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:41:53.635130 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.635049    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xb7c\" (UniqueName: \"kubernetes.io/projected/97aadeef-b4a9-4159-ae79-076e95f95cb4-kube-api-access-4xb7c\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:53.635130 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:53.635079    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aadeef-b4a9-4159-ae79-076e95f95cb4-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:41:54.308358 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:54.308258    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"
Oct 31 07:41:54.308358 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:54.308307    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc" event={"ID":"97aadeef-b4a9-4159-ae79-076e95f95cb4","Type":"ContainerDied","Data":"66a4586817d53e15d94a44a195b55cbab79974f34cfa93527449cd6588ad49ff"}
Oct 31 07:41:54.308558 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:54.308367    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a4586817d53e15d94a44a195b55cbab79974f34cfa93527449cd6588ad49ff"
Oct 31 07:41:57.708229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.708190    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"]
Oct 31 07:41:57.708814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.708799    5829 cpu_manager.go:414] "containerMap: removing container" podUID="97aadeef-b4a9-4159-ae79-076e95f95cb4" containerName="revoke-keys"
Oct 31 07:41:57.708875 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.708815    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97aadeef-b4a9-4159-ae79-076e95f95cb4" containerName="revoke-keys"
Oct 31 07:41:57.708993 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.708981    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="97aadeef-b4a9-4159-ae79-076e95f95cb4" containerName="revoke-keys"
Oct 31 07:41:57.715264 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.715232    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:57.731746 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.731720    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"]
Oct 31 07:41:57.740835 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.740805    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:41:57.741031 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.740832    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:41:57.741031 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.740852    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:41:57.743600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.743582    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:41:57.779846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.779818    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5441a05e-k4kcb\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:57.779998 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.779876    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5441a05e-k4kcb\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:57.779998 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.779983    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xr6\" (UniqueName: \"kubernetes.io/projected/10ce4241-c77c-4a22-8c61-6b6f736a734d-kube-api-access-q7xr6\") pod \"maas-api-revoke-sub-5441a05e-k4kcb\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:57.881266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.881228    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5441a05e-k4kcb\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:57.881266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.881329    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xr6\" (UniqueName: \"kubernetes.io/projected/10ce4241-c77c-4a22-8c61-6b6f736a734d-kube-api-access-q7xr6\") pod \"maas-api-revoke-sub-5441a05e-k4kcb\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:57.881564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.881496    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5441a05e-k4kcb\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:57.881969 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.881942    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5441a05e-k4kcb\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:57.883885 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.883862    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5441a05e-k4kcb\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:57.900788 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:57.900754    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xr6\" (UniqueName: \"kubernetes.io/projected/10ce4241-c77c-4a22-8c61-6b6f736a734d-kube-api-access-q7xr6\") pod \"maas-api-revoke-sub-5441a05e-k4kcb\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:58.030799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:58.030715    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:41:58.382126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:58.382018    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"]
Oct 31 07:41:58.392946 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:41:58.392916    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ce4241_c77c_4a22_8c61_6b6f736a734d.slice/crio-e448995ff83291308a4f9efa2e8f4cf4c737e46958566fdedfbdf415f14bfd02 WatchSource:0}: Error finding container e448995ff83291308a4f9efa2e8f4cf4c737e46958566fdedfbdf415f14bfd02: Status 404 returned error can't find the container with id e448995ff83291308a4f9efa2e8f4cf4c737e46958566fdedfbdf415f14bfd02
Oct 31 07:41:59.351466 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.351426    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb" event={"ID":"10ce4241-c77c-4a22-8c61-6b6f736a734d","Type":"ContainerStarted","Data":"321dfdff9b4afeb5303e41449b061125309ba85d49a22ff3b36f823801f1eb7b"}
Oct 31 07:41:59.351897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.351490    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb" event={"ID":"10ce4241-c77c-4a22-8c61-6b6f736a734d","Type":"ContainerStarted","Data":"e448995ff83291308a4f9efa2e8f4cf4c737e46958566fdedfbdf415f14bfd02"}
Oct 31 07:41:59.391050 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.390984    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb" podStartSLOduration=2.390969266 podStartE2EDuration="2.390969266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:41:59.389450859 +0000 UTC m=+783.528611448" watchObservedRunningTime="2026-10-31 07:41:59.390969266 +0000 UTC m=+783.530129853"
Oct 31 07:41:59.870651 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.870610    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"]
Oct 31 07:41:59.875020 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.874988    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:41:59.896977 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.896919    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"]
Oct 31 07:41:59.897642 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.897615    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc201\"/\"distinct-w3-3cc201-kserve-self-signed-certs\""
Oct 31 07:41:59.919689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.917832    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-home\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:41:59.919689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.917992    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqbml\" (UniqueName: \"kubernetes.io/projected/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-kube-api-access-qqbml\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:41:59.919689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.918114    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tmp-dir\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:41:59.919689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.918197    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-model-cache\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:41:59.919689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.918250    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tls-certs\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:41:59.919689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:41:59.919513    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-dshm\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.020855 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.020818    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-model-cache\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.021075 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.020868    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tls-certs\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.021075 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.020934    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-dshm\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.021075 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.020952    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-home\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.021075 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.021014    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqbml\" (UniqueName: \"kubernetes.io/projected/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-kube-api-access-qqbml\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.021358 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.021134    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tmp-dir\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.021421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.021355    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-model-cache\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.021701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.021672    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-home\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.021807 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.021749    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tmp-dir\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.023649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.023600    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-dshm\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.024292 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.024252    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tls-certs\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.047580 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.047507    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqbml\" (UniqueName: \"kubernetes.io/projected/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-kube-api-access-qqbml\") pod \"distinct-w3-3cc201-kserve-84b76c596b-8lpj9\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.201833 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.201801    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:00.365796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.365743    5829 generic.go:356] "Generic (PLEG): container finished" podID="10ce4241-c77c-4a22-8c61-6b6f736a734d" containerID="321dfdff9b4afeb5303e41449b061125309ba85d49a22ff3b36f823801f1eb7b" exitCode=0
Oct 31 07:42:00.366357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.365987    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb" event={"ID":"10ce4241-c77c-4a22-8c61-6b6f736a734d","Type":"ContainerDied","Data":"321dfdff9b4afeb5303e41449b061125309ba85d49a22ff3b36f823801f1eb7b"}
Oct 31 07:42:00.522518 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.522310    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"]
Oct 31 07:42:00.524497 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:00.524459    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f55dae4_8d1e_4dba_a2d4_b1e791b39157.slice/crio-ca2667a9339a02b10281dcddda69effb4d5aa82cd0cc55554bb05347e6c99ef2 WatchSource:0}: Error finding container ca2667a9339a02b10281dcddda69effb4d5aa82cd0cc55554bb05347e6c99ef2: Status 404 returned error can't find the container with id ca2667a9339a02b10281dcddda69effb4d5aa82cd0cc55554bb05347e6c99ef2
Oct 31 07:42:00.981059 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.981027    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"]
Oct 31 07:42:00.985410 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:00.985392    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.004431 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.004401    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"]
Oct 31 07:42:01.036303 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.036256    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce101d99-vzq94\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.036607 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.036578    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xmr\" (UniqueName: \"kubernetes.io/projected/8fb06198-4ae1-4b31-a462-92e409a7fff1-kube-api-access-d8xmr\") pod \"maas-api-revoke-sub-ce101d99-vzq94\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.036788 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.036770    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce101d99-vzq94\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.138347 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.138316    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce101d99-vzq94\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.138518 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.138415    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce101d99-vzq94\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.138518 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.138449    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xmr\" (UniqueName: \"kubernetes.io/projected/8fb06198-4ae1-4b31-a462-92e409a7fff1-kube-api-access-d8xmr\") pod \"maas-api-revoke-sub-ce101d99-vzq94\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.139123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.139086    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce101d99-vzq94\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.140760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.140738    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce101d99-vzq94\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.157991 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.157964    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xmr\" (UniqueName: \"kubernetes.io/projected/8fb06198-4ae1-4b31-a462-92e409a7fff1-kube-api-access-d8xmr\") pod \"maas-api-revoke-sub-ce101d99-vzq94\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.298776 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.298691    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:01.380362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.380325    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" event={"ID":"3f55dae4-8d1e-4dba-a2d4-b1e791b39157","Type":"ContainerStarted","Data":"fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43"}
Oct 31 07:42:01.380759 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.380377    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" event={"ID":"3f55dae4-8d1e-4dba-a2d4-b1e791b39157","Type":"ContainerStarted","Data":"ca2667a9339a02b10281dcddda69effb4d5aa82cd0cc55554bb05347e6c99ef2"}
Oct 31 07:42:01.420297 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.420228    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podStartSLOduration=2.420211053 podStartE2EDuration="2.420211053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:01.418502198 +0000 UTC m=+785.557662786" watchObservedRunningTime="2026-10-31 07:42:01.420211053 +0000 UTC m=+785.559371640"
Oct 31 07:42:01.507554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.507524    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:42:01.543761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.543731    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-xks-ca\") pod \"10ce4241-c77c-4a22-8c61-6b6f736a734d\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") "
Oct 31 07:42:01.543940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.543869    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-openshift-ca\") pod \"10ce4241-c77c-4a22-8c61-6b6f736a734d\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") "
Oct 31 07:42:01.543940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.543896    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ce4241-c77c-4a22-8c61-6b6f736a734d-kube-api-access-q7xr6\" (UniqueName: \"kubernetes.io/projected/10ce4241-c77c-4a22-8c61-6b6f736a734d-kube-api-access-q7xr6\") pod \"10ce4241-c77c-4a22-8c61-6b6f736a734d\" (UID: \"10ce4241-c77c-4a22-8c61-6b6f736a734d\") "
Oct 31 07:42:01.544276 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.544246    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-openshift-ca" pod "10ce4241-c77c-4a22-8c61-6b6f736a734d" (UID: "10ce4241-c77c-4a22-8c61-6b6f736a734d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:01.544441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.544393    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:01.546574 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.546545    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ce4241-c77c-4a22-8c61-6b6f736a734d-kube-api-access-q7xr6" pod "10ce4241-c77c-4a22-8c61-6b6f736a734d" (UID: "10ce4241-c77c-4a22-8c61-6b6f736a734d"). InnerVolumeSpecName "kube-api-access-q7xr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:01.547080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.547046    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-xks-ca" pod "10ce4241-c77c-4a22-8c61-6b6f736a734d" (UID: "10ce4241-c77c-4a22-8c61-6b6f736a734d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:01.604651 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.604623    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"]
Oct 31 07:42:01.605825 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:01.605795    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb06198_4ae1_4b31_a462_92e409a7fff1.slice/crio-593fc8cfe58181a8caa024e9c8c984404219d091083d9ef933d5df6b4d65b22a WatchSource:0}: Error finding container 593fc8cfe58181a8caa024e9c8c984404219d091083d9ef933d5df6b4d65b22a: Status 404 returned error can't find the container with id 593fc8cfe58181a8caa024e9c8c984404219d091083d9ef933d5df6b4d65b22a
Oct 31 07:42:01.646100 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.646075    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7xr6\" (UniqueName: \"kubernetes.io/projected/10ce4241-c77c-4a22-8c61-6b6f736a734d-kube-api-access-q7xr6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:01.646100 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:01.646101    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ce4241-c77c-4a22-8c61-6b6f736a734d-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:02.201986 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:02.201901    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:02.389720 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:02.389685    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"
Oct 31 07:42:02.389720 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:02.389695    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb" event={"ID":"10ce4241-c77c-4a22-8c61-6b6f736a734d","Type":"ContainerDied","Data":"e448995ff83291308a4f9efa2e8f4cf4c737e46958566fdedfbdf415f14bfd02"}
Oct 31 07:42:02.390243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:02.389750    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e448995ff83291308a4f9efa2e8f4cf4c737e46958566fdedfbdf415f14bfd02"
Oct 31 07:42:02.391819 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:02.391792    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94" event={"ID":"8fb06198-4ae1-4b31-a462-92e409a7fff1","Type":"ContainerStarted","Data":"8f70fabf9bfd8b6faf0b7f6b89b3534b8bb0655f027de519b70ee71dab1d5f01"}
Oct 31 07:42:02.391952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:02.391838    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94" event={"ID":"8fb06198-4ae1-4b31-a462-92e409a7fff1","Type":"ContainerStarted","Data":"593fc8cfe58181a8caa024e9c8c984404219d091083d9ef933d5df6b4d65b22a"}
Oct 31 07:42:02.436588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:02.436539    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94" podStartSLOduration=2.436525048 podStartE2EDuration="2.436525048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:02.432896979 +0000 UTC m=+786.572057568" watchObservedRunningTime="2026-10-31 07:42:02.436525048 +0000 UTC m=+786.575685636"
Oct 31 07:42:03.401116 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.401079    5829 generic.go:356] "Generic (PLEG): container finished" podID="8fb06198-4ae1-4b31-a462-92e409a7fff1" containerID="8f70fabf9bfd8b6faf0b7f6b89b3534b8bb0655f027de519b70ee71dab1d5f01" exitCode=0
Oct 31 07:42:03.401540 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.401157    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94" event={"ID":"8fb06198-4ae1-4b31-a462-92e409a7fff1","Type":"ContainerDied","Data":"8f70fabf9bfd8b6faf0b7f6b89b3534b8bb0655f027de519b70ee71dab1d5f01"}
Oct 31 07:42:03.858220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.858135    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"]
Oct 31 07:42:03.858827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.858810    5829 cpu_manager.go:414] "containerMap: removing container" podUID="10ce4241-c77c-4a22-8c61-6b6f736a734d" containerName="revoke-keys"
Oct 31 07:42:03.858869 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.858829    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ce4241-c77c-4a22-8c61-6b6f736a734d" containerName="revoke-keys"
Oct 31 07:42:03.858955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.858946    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ce4241-c77c-4a22-8c61-6b6f736a734d" containerName="revoke-keys"
Oct 31 07:42:03.862565 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.862549    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:03.878987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.878961    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4794f5-openshift-default-dockercfg-lqf5r\""
Oct 31 07:42:03.895554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.895526    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"]
Oct 31 07:42:03.976992 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.976961    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a327099c-8eca-483f-b558-f36f1edb0d97-istiod-ca-cert\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:03.977163 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.977005    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljm2\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-kube-api-access-tljm2\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:03.977163 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.977063    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:03.977163 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.977096    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-data\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:03.977272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.977180    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-envoy\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:03.977272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.977235    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a327099c-8eca-483f-b558-f36f1edb0d97-istio-podinfo\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:03.977365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.977293    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-credential-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:03.977396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.977363    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-certs\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:03.977429 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:03.977417    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-istio-token\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.078727 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.078683    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a327099c-8eca-483f-b558-f36f1edb0d97-istiod-ca-cert\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.078928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.078736    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljm2\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-kube-api-access-tljm2\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.078928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.078763    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.078928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.078780    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-data\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.078928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.078813    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-envoy\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.078928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.078854    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a327099c-8eca-483f-b558-f36f1edb0d97-istio-podinfo\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.078928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.078900    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-credential-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.079206 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.078935    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-certs\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.079206 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.079059    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-istio-token\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.079353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.079240    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.079353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.079322    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-data\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.079566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.079541    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a327099c-8eca-483f-b558-f36f1edb0d97-istiod-ca-cert\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.079640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.079581    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-certs\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.079640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.079602    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-credential-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.081318 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.081276    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-envoy\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.081741 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.081719    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a327099c-8eca-483f-b558-f36f1edb0d97-istio-podinfo\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.099633 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.099603    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-istio-token\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.099779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.099655    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljm2\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-kube-api-access-tljm2\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.176693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.176658    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:04.508721 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.505525    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"]
Oct 31 07:42:04.518207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.518168    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:42:04.518343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.518246    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:42:04.518343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.518273    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:42:04.566857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.566824    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:04.686361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.686326    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb06198-4ae1-4b31-a462-92e409a7fff1-kube-api-access-d8xmr\" (UniqueName: \"kubernetes.io/projected/8fb06198-4ae1-4b31-a462-92e409a7fff1-kube-api-access-d8xmr\") pod \"8fb06198-4ae1-4b31-a462-92e409a7fff1\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") "
Oct 31 07:42:04.686545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.686521    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-openshift-ca\") pod \"8fb06198-4ae1-4b31-a462-92e409a7fff1\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") "
Oct 31 07:42:04.686644 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.686630    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-xks-ca\") pod \"8fb06198-4ae1-4b31-a462-92e409a7fff1\" (UID: \"8fb06198-4ae1-4b31-a462-92e409a7fff1\") "
Oct 31 07:42:04.686888 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.686859    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-openshift-ca" pod "8fb06198-4ae1-4b31-a462-92e409a7fff1" (UID: "8fb06198-4ae1-4b31-a462-92e409a7fff1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:04.687182 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.687161    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:04.688638 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.688606    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-xks-ca" pod "8fb06198-4ae1-4b31-a462-92e409a7fff1" (UID: "8fb06198-4ae1-4b31-a462-92e409a7fff1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:04.688874 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.688852    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb06198-4ae1-4b31-a462-92e409a7fff1-kube-api-access-d8xmr" pod "8fb06198-4ae1-4b31-a462-92e409a7fff1" (UID: "8fb06198-4ae1-4b31-a462-92e409a7fff1"). InnerVolumeSpecName "kube-api-access-d8xmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:04.788905 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.788821    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8xmr\" (UniqueName: \"kubernetes.io/projected/8fb06198-4ae1-4b31-a462-92e409a7fff1-kube-api-access-d8xmr\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:04.788905 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:04.788853    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb06198-4ae1-4b31-a462-92e409a7fff1-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:05.432261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:05.432224    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"
Oct 31 07:42:05.432261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:05.432252    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94" event={"ID":"8fb06198-4ae1-4b31-a462-92e409a7fff1","Type":"ContainerDied","Data":"593fc8cfe58181a8caa024e9c8c984404219d091083d9ef933d5df6b4d65b22a"}
Oct 31 07:42:05.432928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:05.432322    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="593fc8cfe58181a8caa024e9c8c984404219d091083d9ef933d5df6b4d65b22a"
Oct 31 07:42:05.434147 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:05.434122    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" event={"ID":"a327099c-8eca-483f-b558-f36f1edb0d97","Type":"ContainerStarted","Data":"547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3"}
Oct 31 07:42:05.434267 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:05.434170    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" event={"ID":"a327099c-8eca-483f-b558-f36f1edb0d97","Type":"ContainerStarted","Data":"d8937ba19dc678b036fd0f85bd1a80dbb8cacecb14d2fd5fe55ed0e4ed7350eb"}
Oct 31 07:42:05.502687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:05.502636    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" podStartSLOduration=2.502616829 podStartE2EDuration="2.502616829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:05.487913287 +0000 UTC m=+789.627073874" watchObservedRunningTime="2026-10-31 07:42:05.502616829 +0000 UTC m=+789.641777416"
Oct 31 07:42:06.177829 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:06.177797    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:06.179438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:06.179412    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 07:42:06.179559 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:06.179460    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 07:42:07.177269 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:07.177232    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 07:42:07.177479 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:07.177309    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 07:42:08.177687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:08.177650    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 07:42:08.178091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:08.177711    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 07:42:09.177387 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:09.177351    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 07:42:09.177577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:09.177407    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 07:42:09.844188 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:09.844149    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:42:10.177690 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:10.177655    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 07:42:10.177873 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:10.177723    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 07:42:10.202521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:10.202493    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:10.244647 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:10.244611    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:10.479829 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:10.479745    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:42:11.791313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.784061    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:11.824305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.816429    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:11.852307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.842344    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:42:11.901646 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.901341    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"]
Oct 31 07:42:11.901969 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.901953    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8fb06198-4ae1-4b31-a462-92e409a7fff1" containerName="revoke-keys"
Oct 31 07:42:11.901969 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.901970    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb06198-4ae1-4b31-a462-92e409a7fff1" containerName="revoke-keys"
Oct 31 07:42:11.902139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.902089    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb06198-4ae1-4b31-a462-92e409a7fff1" containerName="revoke-keys"
Oct 31 07:42:11.912377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.911454    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"]
Oct 31 07:42:11.912377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.911576    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:11.927423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.926914    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:42:11.927527 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.927446    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:42:11.927527 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.927510    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:42:11.927644 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:11.927572    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:42:12.056705 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.056604    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20bfd47-9dbnh\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.057225 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.057196    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20bfd47-9dbnh\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.057524 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.057505    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgpg\" (UniqueName: \"kubernetes.io/projected/a7f9597b-20d1-4022-bd76-e9942fdd88ff-kube-api-access-rqgpg\") pod \"maas-api-revoke-sub-d20bfd47-9dbnh\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.158488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.158452    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20bfd47-9dbnh\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.158674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.158553    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgpg\" (UniqueName: \"kubernetes.io/projected/a7f9597b-20d1-4022-bd76-e9942fdd88ff-kube-api-access-rqgpg\") pod \"maas-api-revoke-sub-d20bfd47-9dbnh\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.158674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.158597    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20bfd47-9dbnh\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.159339 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.159313    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d20bfd47-9dbnh\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.161072 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.161046    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d20bfd47-9dbnh\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.182206 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.182176    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgpg\" (UniqueName: \"kubernetes.io/projected/a7f9597b-20d1-4022-bd76-e9942fdd88ff-kube-api-access-rqgpg\") pod \"maas-api-revoke-sub-d20bfd47-9dbnh\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.247670 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.247633    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:12.557848 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.557784    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"]
Oct 31 07:42:12.558502 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:12.558468    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f9597b_20d1_4022_bd76_e9942fdd88ff.slice/crio-53b01f12f8983c216236c9a7fa054f42e85e8557601d9d68a272c66053a4d3bd WatchSource:0}: Error finding container 53b01f12f8983c216236c9a7fa054f42e85e8557601d9d68a272c66053a4d3bd: Status 404 returned error can't find the container with id 53b01f12f8983c216236c9a7fa054f42e85e8557601d9d68a272c66053a4d3bd
Oct 31 07:42:12.633524 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:12.633483    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh" event={"ID":"a7f9597b-20d1-4022-bd76-e9942fdd88ff","Type":"ContainerStarted","Data":"53b01f12f8983c216236c9a7fa054f42e85e8557601d9d68a272c66053a4d3bd"}
Oct 31 07:42:13.643105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:13.643065    5829 generic.go:356] "Generic (PLEG): container finished" podID="a7f9597b-20d1-4022-bd76-e9942fdd88ff" containerID="e9d7e972d1c7e1115bfbf5d76959d54fcb62756180512164270d23b7ba90a1ec" exitCode=0
Oct 31 07:42:13.643622 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:13.643151    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh" event={"ID":"a7f9597b-20d1-4022-bd76-e9942fdd88ff","Type":"ContainerDied","Data":"e9d7e972d1c7e1115bfbf5d76959d54fcb62756180512164270d23b7ba90a1ec"}
Oct 31 07:42:14.814390 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:14.814365    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:14.998555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:14.998456    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-openshift-ca\") pod \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") "
Oct 31 07:42:14.998555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:14.998504    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f9597b-20d1-4022-bd76-e9942fdd88ff-kube-api-access-rqgpg\" (UniqueName: \"kubernetes.io/projected/a7f9597b-20d1-4022-bd76-e9942fdd88ff-kube-api-access-rqgpg\") pod \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") "
Oct 31 07:42:14.998746 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:14.998632    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-xks-ca\") pod \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\" (UID: \"a7f9597b-20d1-4022-bd76-e9942fdd88ff\") "
Oct 31 07:42:14.998935 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:14.998906    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-openshift-ca" pod "a7f9597b-20d1-4022-bd76-e9942fdd88ff" (UID: "a7f9597b-20d1-4022-bd76-e9942fdd88ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:14.999062 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:14.999047    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:15.000784 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:15.000757    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-xks-ca" pod "a7f9597b-20d1-4022-bd76-e9942fdd88ff" (UID: "a7f9597b-20d1-4022-bd76-e9942fdd88ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:15.000905 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:15.000786    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f9597b-20d1-4022-bd76-e9942fdd88ff-kube-api-access-rqgpg" pod "a7f9597b-20d1-4022-bd76-e9942fdd88ff" (UID: "a7f9597b-20d1-4022-bd76-e9942fdd88ff"). InnerVolumeSpecName "kube-api-access-rqgpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:15.100571 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:15.100521    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqgpg\" (UniqueName: \"kubernetes.io/projected/a7f9597b-20d1-4022-bd76-e9942fdd88ff-kube-api-access-rqgpg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:15.100571 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:15.100565    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f9597b-20d1-4022-bd76-e9942fdd88ff-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:15.658761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:15.658730    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"
Oct 31 07:42:15.658926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:15.658771    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh" event={"ID":"a7f9597b-20d1-4022-bd76-e9942fdd88ff","Type":"ContainerDied","Data":"53b01f12f8983c216236c9a7fa054f42e85e8557601d9d68a272c66053a4d3bd"}
Oct 31 07:42:15.658926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:15.658831    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53b01f12f8983c216236c9a7fa054f42e85e8557601d9d68a272c66053a4d3bd"
Oct 31 07:42:16.511158 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:42:16.511119    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:42:18.442693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.442650    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"]
Oct 31 07:42:18.443371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.443353    5829 cpu_manager.go:414] "containerMap: removing container" podUID="a7f9597b-20d1-4022-bd76-e9942fdd88ff" containerName="revoke-keys"
Oct 31 07:42:18.443457 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.443373    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f9597b-20d1-4022-bd76-e9942fdd88ff" containerName="revoke-keys"
Oct 31 07:42:18.443513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.443503    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f9597b-20d1-4022-bd76-e9942fdd88ff" containerName="revoke-keys"
Oct 31 07:42:18.447312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.447292    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.462831 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.462808    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-b4e8fd-openshift-default-dockercfg-v2jcn\""
Oct 31 07:42:18.479815 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.479780    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"]
Oct 31 07:42:18.540381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.540347    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-certs\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.540554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.540408    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab464c-8519-4d3d-9d97-10cade920d34-istiod-ca-cert\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.540554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.540475    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-data\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.540640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.540600    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm2s8\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-kube-api-access-cm2s8\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.540675 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.540655    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.540717 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.540685    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-credential-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.540763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.540718    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-token\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.540808 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.540776    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-envoy\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.540843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.540827    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-podinfo\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.641823 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.641783    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.641833    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-credential-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.641868    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-token\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.641908    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-envoy\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.641938    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-podinfo\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.641980    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-certs\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.642023    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab464c-8519-4d3d-9d97-10cade920d34-istiod-ca-cert\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642420 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.642052    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-data\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642420 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.642166    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm2s8\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-kube-api-access-cm2s8\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642420 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.642333    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642420 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.642387    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-credential-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642644 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.642453    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-certs\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642644 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.642621    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-data\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.642833 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.642807    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab464c-8519-4d3d-9d97-10cade920d34-istiod-ca-cert\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.644414 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.644392    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-envoy\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.644631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.644605    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-podinfo\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.664779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.664740    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-token\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.664889 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.664859    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm2s8\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-kube-api-access-cm2s8\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.761764 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.761675    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:18.813854 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.813814    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"]
Oct 31 07:42:18.819557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.819524    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.845450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.844969    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-certs\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.845450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.845022    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-credential-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.845450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.845066    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-data\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.845450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.845163    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49d057-fed8-4675-8272-8da07af38b1d-istiod-ca-cert\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.845450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.845202    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-envoy\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.845450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.845225    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-istio-token\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.845450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.845268    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95wbv\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-kube-api-access-95wbv\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.846266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.846023    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49d057-fed8-4675-8272-8da07af38b1d-istio-podinfo\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.846266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.846132    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.847885 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.847716    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"]
Oct 31 07:42:18.947025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.946979    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-data\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947085    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49d057-fed8-4675-8272-8da07af38b1d-istiod-ca-cert\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947124    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-envoy\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947144    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-istio-token\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947186    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95wbv\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-kube-api-access-95wbv\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947219    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49d057-fed8-4675-8272-8da07af38b1d-istio-podinfo\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947541 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947350    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947541 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947465    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-certs\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947541 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947498    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-credential-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947464    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-data\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947756 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947726    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947839 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947811    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-certs\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947900 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947864    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49d057-fed8-4675-8272-8da07af38b1d-istiod-ca-cert\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.947949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.947898    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-credential-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.950054 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.950032    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-envoy\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.950239 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.950223    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49d057-fed8-4675-8272-8da07af38b1d-istio-podinfo\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.968052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.968028    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-istio-token\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:18.968183 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:18.968133    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95wbv\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-kube-api-access-95wbv\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:19.092106 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.092063    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"]
Oct 31 07:42:19.092860 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:19.092835    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ab464c_8519_4d3d_9d97_10cade920d34.slice/crio-4f9b36643e8fa46132a67dffd262ab89fc6593d737dc1919d4c6321d9f68e376 WatchSource:0}: Error finding container 4f9b36643e8fa46132a67dffd262ab89fc6593d737dc1919d4c6321d9f68e376: Status 404 returned error can't find the container with id 4f9b36643e8fa46132a67dffd262ab89fc6593d737dc1919d4c6321d9f68e376
Oct 31 07:42:19.095496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.095463    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:42:19.095586 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.095551    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:42:19.095654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.095597    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:42:19.149821 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.149787    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:19.483160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.483126    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"]
Oct 31 07:42:19.484365 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:19.484335    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a49d057_fed8_4675_8272_8da07af38b1d.slice/crio-09b03ebe71775dbb63384bb34233566ea44e0e6332bed37cb7fb2a10adf7adf7 WatchSource:0}: Error finding container 09b03ebe71775dbb63384bb34233566ea44e0e6332bed37cb7fb2a10adf7adf7: Status 404 returned error can't find the container with id 09b03ebe71775dbb63384bb34233566ea44e0e6332bed37cb7fb2a10adf7adf7
Oct 31 07:42:19.487320 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.487265    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:42:19.487399 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.487379    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:42:19.487445 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.487424    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:42:19.700241 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.700194    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" event={"ID":"d7ab464c-8519-4d3d-9d97-10cade920d34","Type":"ContainerStarted","Data":"05ae96899eee0995d9d499d4b34f7743f78a2672c5e707820b3eafa60357a7da"}
Oct 31 07:42:19.700451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.700268    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" event={"ID":"d7ab464c-8519-4d3d-9d97-10cade920d34","Type":"ContainerStarted","Data":"4f9b36643e8fa46132a67dffd262ab89fc6593d737dc1919d4c6321d9f68e376"}
Oct 31 07:42:19.704537 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.703469    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" event={"ID":"1a49d057-fed8-4675-8272-8da07af38b1d","Type":"ContainerStarted","Data":"949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f"}
Oct 31 07:42:19.704537 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.703528    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" event={"ID":"1a49d057-fed8-4675-8272-8da07af38b1d","Type":"ContainerStarted","Data":"09b03ebe71775dbb63384bb34233566ea44e0e6332bed37cb7fb2a10adf7adf7"}
Oct 31 07:42:19.755889 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.755770    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podStartSLOduration=1.755753817 podStartE2EDuration="1.755753817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:19.755487147 +0000 UTC m=+803.894647734" watchObservedRunningTime="2026-10-31 07:42:19.755753817 +0000 UTC m=+803.894914405"
Oct 31 07:42:19.761785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.761757    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:19.798350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:19.798260    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" podStartSLOduration=1.7982449790000001 podStartE2EDuration="1.798244979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:19.796144575 +0000 UTC m=+803.935305182" watchObservedRunningTime="2026-10-31 07:42:19.798244979 +0000 UTC m=+803.937405567"
Oct 31 07:42:20.152163 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.152132    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:20.157423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.157376    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 07:42:20.157596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.157460    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 07:42:20.675015 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.674974    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"]
Oct 31 07:42:20.679489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.679470    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.698243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.698205    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"]
Oct 31 07:42:20.725739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.725703    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:42:20.726723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.726696    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:42:20.757306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.733552    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:42:20.757584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.750514    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:42:20.763515 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.763481    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 07:42:20.763788 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.763757    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 07:42:20.771444 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.771408    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7e07fc7-lhc2b\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.771678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.771514    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdd5l\" (UniqueName: \"kubernetes.io/projected/79f0aaaa-0edc-4427-938c-cd401a65e0a5-kube-api-access-fdd5l\") pod \"maas-api-revoke-sub-d7e07fc7-lhc2b\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.771678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.771605    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7e07fc7-lhc2b\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.874557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.873266    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdd5l\" (UniqueName: \"kubernetes.io/projected/79f0aaaa-0edc-4427-938c-cd401a65e0a5-kube-api-access-fdd5l\") pod \"maas-api-revoke-sub-d7e07fc7-lhc2b\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.874557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.873423    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7e07fc7-lhc2b\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.874557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.873726    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7e07fc7-lhc2b\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.875148 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.875021    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7e07fc7-lhc2b\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.877779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.877753    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7e07fc7-lhc2b\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.894443 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.894421    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdd5l\" (UniqueName: \"kubernetes.io/projected/79f0aaaa-0edc-4427-938c-cd401a65e0a5-kube-api-access-fdd5l\") pod \"maas-api-revoke-sub-d7e07fc7-lhc2b\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:20.995409 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:20.995333    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:21.161816 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:21.152690    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 07:42:21.161816 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:21.152750    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 07:42:21.363401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:21.363371    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"]
Oct 31 07:42:21.366875 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:21.366370    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f0aaaa_0edc_4427_938c_cd401a65e0a5.slice/crio-ba83da5a936ea9302b2684178fc6552444b8db5090848bbf24b9936ae4c71416 WatchSource:0}: Error finding container ba83da5a936ea9302b2684178fc6552444b8db5090848bbf24b9936ae4c71416: Status 404 returned error can't find the container with id ba83da5a936ea9302b2684178fc6552444b8db5090848bbf24b9936ae4c71416
Oct 31 07:42:21.762983 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:21.762941    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 07:42:21.763494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:21.763004    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 07:42:21.782616 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:21.782579    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b" event={"ID":"79f0aaaa-0edc-4427-938c-cd401a65e0a5","Type":"ContainerStarted","Data":"ba83da5a936ea9302b2684178fc6552444b8db5090848bbf24b9936ae4c71416"}
Oct 31 07:42:22.153261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:22.152743    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 07:42:22.153261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:22.152816    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 07:42:22.762216 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:22.762172    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 07:42:22.762421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:22.762237    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 07:42:22.792208 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:22.792174    5829 generic.go:356] "Generic (PLEG): container finished" podID="79f0aaaa-0edc-4427-938c-cd401a65e0a5" containerID="72481dcbb84bcaee5774395716f713b7e613db51d6b507c13678eed5db9ff44f" exitCode=0
Oct 31 07:42:22.792602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:22.792261    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b" event={"ID":"79f0aaaa-0edc-4427-938c-cd401a65e0a5","Type":"ContainerDied","Data":"72481dcbb84bcaee5774395716f713b7e613db51d6b507c13678eed5db9ff44f"}
Oct 31 07:42:23.154181 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:23.153737    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 07:42:23.154181 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:23.153791    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 07:42:23.762150 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:23.762114    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 07:42:23.762356 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:23.762165    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 07:42:24.155100 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.153779    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 07:42:24.155100 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.153843    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 07:42:24.157699 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.157672    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:24.268060 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.268020    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-xks-ca\") pod \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") "
Oct 31 07:42:24.268294 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.268166    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-openshift-ca\") pod \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") "
Oct 31 07:42:24.268294 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.268221    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f0aaaa-0edc-4427-938c-cd401a65e0a5-kube-api-access-fdd5l\" (UniqueName: \"kubernetes.io/projected/79f0aaaa-0edc-4427-938c-cd401a65e0a5-kube-api-access-fdd5l\") pod \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\" (UID: \"79f0aaaa-0edc-4427-938c-cd401a65e0a5\") "
Oct 31 07:42:24.268963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.268588    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-openshift-ca" pod "79f0aaaa-0edc-4427-938c-cd401a65e0a5" (UID: "79f0aaaa-0edc-4427-938c-cd401a65e0a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:24.268963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.268931    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:24.271478 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.271453    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-xks-ca" pod "79f0aaaa-0edc-4427-938c-cd401a65e0a5" (UID: "79f0aaaa-0edc-4427-938c-cd401a65e0a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:24.271593 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.271461    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f0aaaa-0edc-4427-938c-cd401a65e0a5-kube-api-access-fdd5l" pod "79f0aaaa-0edc-4427-938c-cd401a65e0a5" (UID: "79f0aaaa-0edc-4427-938c-cd401a65e0a5"). InnerVolumeSpecName "kube-api-access-fdd5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:24.372091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.370727    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79f0aaaa-0edc-4427-938c-cd401a65e0a5-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:24.372091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.370765    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdd5l\" (UniqueName: \"kubernetes.io/projected/79f0aaaa-0edc-4427-938c-cd401a65e0a5-kube-api-access-fdd5l\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:24.762117 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.762084    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 07:42:24.762310 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.762134    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 07:42:24.854587 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.854530    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b" event={"ID":"79f0aaaa-0edc-4427-938c-cd401a65e0a5","Type":"ContainerDied","Data":"ba83da5a936ea9302b2684178fc6552444b8db5090848bbf24b9936ae4c71416"}
Oct 31 07:42:24.854780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.854645    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba83da5a936ea9302b2684178fc6552444b8db5090848bbf24b9936ae4c71416"
Oct 31 07:42:24.854780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:24.854719    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"
Oct 31 07:42:25.151074 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.151040    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 07:42:25.151291 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.151140    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 07:42:25.762645 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.762601    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 07:42:25.763051 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.762691    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 07:42:25.870763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.870726    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"]
Oct 31 07:42:25.871765 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.871743    5829 cpu_manager.go:414] "containerMap: removing container" podUID="79f0aaaa-0edc-4427-938c-cd401a65e0a5" containerName="revoke-keys"
Oct 31 07:42:25.871837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.871769    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f0aaaa-0edc-4427-938c-cd401a65e0a5" containerName="revoke-keys"
Oct 31 07:42:25.871986 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.871971    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f0aaaa-0edc-4427-938c-cd401a65e0a5" containerName="revoke-keys"
Oct 31 07:42:25.876753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.876724    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:25.900712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.900505    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:42:25.900712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.900590    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:42:25.900712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.900677    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:42:25.901229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.901205    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"]
Oct 31 07:42:25.908306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:25.906496    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:42:26.000611 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.000574    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn2kt\" (UniqueName: \"kubernetes.io/projected/ad412fdc-281e-4558-88d7-56960f953b45-kube-api-access-pn2kt\") pod \"maas-api-revoke-sub-b0a3ab8e-ksl98\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.000795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.000616    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a3ab8e-ksl98\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.000795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.000654    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a3ab8e-ksl98\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.102876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.102746    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn2kt\" (UniqueName: \"kubernetes.io/projected/ad412fdc-281e-4558-88d7-56960f953b45-kube-api-access-pn2kt\") pod \"maas-api-revoke-sub-b0a3ab8e-ksl98\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.102876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.102810    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a3ab8e-ksl98\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.102876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.102854    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a3ab8e-ksl98\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.103638 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.103615    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0a3ab8e-ksl98\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.106231 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.106202    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0a3ab8e-ksl98\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.122986 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.122952    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn2kt\" (UniqueName: \"kubernetes.io/projected/ad412fdc-281e-4558-88d7-56960f953b45-kube-api-access-pn2kt\") pod \"maas-api-revoke-sub-b0a3ab8e-ksl98\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.213612 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.213574    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:26.459306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.458428    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:26.460924 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.460898    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:26.465922 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.463914    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:42:26.547644 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.547614    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"]
Oct 31 07:42:26.548989 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:26.548957    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad412fdc_281e_4558_88d7_56960f953b45.slice/crio-edd981aa4f184398dbece02737d6d9c928686dad8148d0a9cb9cba13797aef07 WatchSource:0}: Error finding container edd981aa4f184398dbece02737d6d9c928686dad8148d0a9cb9cba13797aef07: Status 404 returned error can't find the container with id edd981aa4f184398dbece02737d6d9c928686dad8148d0a9cb9cba13797aef07
Oct 31 07:42:26.766645 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.766614    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:26.767104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.767087    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:26.767733 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.767718    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:42:26.894123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.894077    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98" event={"ID":"ad412fdc-281e-4558-88d7-56960f953b45","Type":"ContainerStarted","Data":"8e1ab5cca387dbbe84b289bd178f5a4c79c94b414c3b88ae6c239713a5de87ae"}
Oct 31 07:42:26.894411 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.894183    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98" event={"ID":"ad412fdc-281e-4558-88d7-56960f953b45","Type":"ContainerStarted","Data":"edd981aa4f184398dbece02737d6d9c928686dad8148d0a9cb9cba13797aef07"}
Oct 31 07:42:26.935317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:26.935244    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98" podStartSLOduration=1.935227835 podStartE2EDuration="1.935227835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:26.933425021 +0000 UTC m=+811.072585606" watchObservedRunningTime="2026-10-31 07:42:26.935227835 +0000 UTC m=+811.074388423"
Oct 31 07:42:27.903827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:27.903787    5829 generic.go:356] "Generic (PLEG): container finished" podID="ad412fdc-281e-4558-88d7-56960f953b45" containerID="8e1ab5cca387dbbe84b289bd178f5a4c79c94b414c3b88ae6c239713a5de87ae" exitCode=0
Oct 31 07:42:27.904333 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:27.903876    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98" event={"ID":"ad412fdc-281e-4558-88d7-56960f953b45","Type":"ContainerDied","Data":"8e1ab5cca387dbbe84b289bd178f5a4c79c94b414c3b88ae6c239713a5de87ae"}
Oct 31 07:42:29.469946 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.469918    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:29.593372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.586098    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-xks-ca\") pod \"ad412fdc-281e-4558-88d7-56960f953b45\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") "
Oct 31 07:42:29.593372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.586159    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad412fdc-281e-4558-88d7-56960f953b45-kube-api-access-pn2kt\" (UniqueName: \"kubernetes.io/projected/ad412fdc-281e-4558-88d7-56960f953b45-kube-api-access-pn2kt\") pod \"ad412fdc-281e-4558-88d7-56960f953b45\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") "
Oct 31 07:42:29.593372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.586230    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-openshift-ca\") pod \"ad412fdc-281e-4558-88d7-56960f953b45\" (UID: \"ad412fdc-281e-4558-88d7-56960f953b45\") "
Oct 31 07:42:29.610237 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.608872    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-openshift-ca" pod "ad412fdc-281e-4558-88d7-56960f953b45" (UID: "ad412fdc-281e-4558-88d7-56960f953b45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:29.727026 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.725419    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:29.746458 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.746399    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-xks-ca" pod "ad412fdc-281e-4558-88d7-56960f953b45" (UID: "ad412fdc-281e-4558-88d7-56960f953b45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:29.746935 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.746909    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad412fdc-281e-4558-88d7-56960f953b45-kube-api-access-pn2kt" pod "ad412fdc-281e-4558-88d7-56960f953b45" (UID: "ad412fdc-281e-4558-88d7-56960f953b45"). InnerVolumeSpecName "kube-api-access-pn2kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:29.828082 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.827173    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad412fdc-281e-4558-88d7-56960f953b45-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:29.828082 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:29.827213    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn2kt\" (UniqueName: \"kubernetes.io/projected/ad412fdc-281e-4558-88d7-56960f953b45-kube-api-access-pn2kt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:30.021404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:30.021359    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"
Oct 31 07:42:30.053671 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:30.053626    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98" event={"ID":"ad412fdc-281e-4558-88d7-56960f953b45","Type":"ContainerDied","Data":"edd981aa4f184398dbece02737d6d9c928686dad8148d0a9cb9cba13797aef07"}
Oct 31 07:42:30.081403 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:30.081372    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edd981aa4f184398dbece02737d6d9c928686dad8148d0a9cb9cba13797aef07"
Oct 31 07:42:34.345256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.345219    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"]
Oct 31 07:42:34.345914 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.345896    5829 cpu_manager.go:414] "containerMap: removing container" podUID="ad412fdc-281e-4558-88d7-56960f953b45" containerName="revoke-keys"
Oct 31 07:42:34.345914 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.345913    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad412fdc-281e-4558-88d7-56960f953b45" containerName="revoke-keys"
Oct 31 07:42:34.346053 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.346001    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad412fdc-281e-4558-88d7-56960f953b45" containerName="revoke-keys"
Oct 31 07:42:34.351136 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.351114    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.364143 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.364114    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"]
Oct 31 07:42:34.370991 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.370536    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.375903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.375877    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"]
Oct 31 07:42:34.389081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.386515    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:42:34.389081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.386579    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3dff3f\"/\"distinct-2-w4-3dff3f-kserve-self-signed-certs\""
Oct 31 07:42:34.389081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.386730    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:42:34.391426 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.391391    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-home\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.392410 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.392383    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f79k\" (UniqueName: \"kubernetes.io/projected/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-kube-api-access-8f79k\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.392613 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.392597    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-dshm\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.392800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.392779    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tls-certs\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.392942 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.392925    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8880ccd8-grpwx\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.393086 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.393069    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8880ccd8-grpwx\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.393245 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.393228    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-model-cache\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.394589 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.394569    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptnq4\" (UniqueName: \"kubernetes.io/projected/0d202dea-f0ad-4fe1-9b03-be291a9b121a-kube-api-access-ptnq4\") pod \"maas-api-revoke-sub-8880ccd8-grpwx\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.394827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.394320    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"]
Oct 31 07:42:34.394952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.394925    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tmp-dir\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.467241 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.467215    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:42:34.467241 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.467235    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:42:34.496207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496172    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-home\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.496401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496227    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f79k\" (UniqueName: \"kubernetes.io/projected/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-kube-api-access-8f79k\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.496401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496276    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-dshm\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.496401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496308    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tls-certs\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.496401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496326    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8880ccd8-grpwx\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.496401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496346    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8880ccd8-grpwx\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.496401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496374    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-model-cache\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.496717 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496447    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptnq4\" (UniqueName: \"kubernetes.io/projected/0d202dea-f0ad-4fe1-9b03-be291a9b121a-kube-api-access-ptnq4\") pod \"maas-api-revoke-sub-8880ccd8-grpwx\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.496717 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496543    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tmp-dir\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.496717 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496695    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-home\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.496874 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496808    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-model-cache\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.496930 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.496909    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tmp-dir\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.497322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.497298    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8880ccd8-grpwx\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.498960 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.498941    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tls-certs\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.499134 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.499111    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-dshm\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.499239 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.499218    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8880ccd8-grpwx\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.517258 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.517224    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f79k\" (UniqueName: \"kubernetes.io/projected/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-kube-api-access-8f79k\") pod \"distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") " pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.517258 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.517243    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptnq4\" (UniqueName: \"kubernetes.io/projected/0d202dea-f0ad-4fe1-9b03-be291a9b121a-kube-api-access-ptnq4\") pod \"maas-api-revoke-sub-8880ccd8-grpwx\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.676807 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.676774    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:34.705908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.705874    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:34.987443 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:34.987413    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"]
Oct 31 07:42:34.988468 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:34.988436    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8bd0c5e_5c83_4734_bf2b_ff2efb34d2d5.slice/crio-d2973c47118767416b18a1d19b6601fa503f746acde92838d85052ff06218024 WatchSource:0}: Error finding container d2973c47118767416b18a1d19b6601fa503f746acde92838d85052ff06218024: Status 404 returned error can't find the container with id d2973c47118767416b18a1d19b6601fa503f746acde92838d85052ff06218024
Oct 31 07:42:35.026007 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:35.025984    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"]
Oct 31 07:42:35.028130 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:35.028090    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d202dea_f0ad_4fe1_9b03_be291a9b121a.slice/crio-35fc2fd33024d3cc22d9500e6c41540702f51fc620986ef2e2ad39a9ea1485e9 WatchSource:0}: Error finding container 35fc2fd33024d3cc22d9500e6c41540702f51fc620986ef2e2ad39a9ea1485e9: Status 404 returned error can't find the container with id 35fc2fd33024d3cc22d9500e6c41540702f51fc620986ef2e2ad39a9ea1485e9
Oct 31 07:42:35.052484 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:35.052450    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx" event={"ID":"0d202dea-f0ad-4fe1-9b03-be291a9b121a","Type":"ContainerStarted","Data":"35fc2fd33024d3cc22d9500e6c41540702f51fc620986ef2e2ad39a9ea1485e9"}
Oct 31 07:42:35.053940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:35.053897    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" event={"ID":"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5","Type":"ContainerStarted","Data":"d2973c47118767416b18a1d19b6601fa503f746acde92838d85052ff06218024"}
Oct 31 07:42:36.063087 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:36.063053    5829 generic.go:356] "Generic (PLEG): container finished" podID="0d202dea-f0ad-4fe1-9b03-be291a9b121a" containerID="29999e2749e1d18bfcd5520aeda06fd5a0f5a443a7dd5bc1ff073ee1a07573ec" exitCode=0
Oct 31 07:42:36.063528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:36.063128    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx" event={"ID":"0d202dea-f0ad-4fe1-9b03-be291a9b121a","Type":"ContainerDied","Data":"29999e2749e1d18bfcd5520aeda06fd5a0f5a443a7dd5bc1ff073ee1a07573ec"}
Oct 31 07:42:36.064798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:36.064766    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" event={"ID":"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5","Type":"ContainerStarted","Data":"2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376"}
Oct 31 07:42:36.144496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:36.144440    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podStartSLOduration=2.144425049 podStartE2EDuration="2.144425049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:36.139554249 +0000 UTC m=+820.278714857" watchObservedRunningTime="2026-10-31 07:42:36.144425049 +0000 UTC m=+820.283585636"
Oct 31 07:42:36.677214 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:36.677180    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:37.248704 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.248676    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:37.336312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.336211    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d202dea-f0ad-4fe1-9b03-be291a9b121a-kube-api-access-ptnq4\" (UniqueName: \"kubernetes.io/projected/0d202dea-f0ad-4fe1-9b03-be291a9b121a-kube-api-access-ptnq4\") pod \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") "
Oct 31 07:42:37.336565 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.336322    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-xks-ca\") pod \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") "
Oct 31 07:42:37.336565 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.336397    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-openshift-ca\") pod \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\" (UID: \"0d202dea-f0ad-4fe1-9b03-be291a9b121a\") "
Oct 31 07:42:37.336777 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.336747    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-openshift-ca" pod "0d202dea-f0ad-4fe1-9b03-be291a9b121a" (UID: "0d202dea-f0ad-4fe1-9b03-be291a9b121a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:37.338432 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.338405    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d202dea-f0ad-4fe1-9b03-be291a9b121a-kube-api-access-ptnq4" pod "0d202dea-f0ad-4fe1-9b03-be291a9b121a" (UID: "0d202dea-f0ad-4fe1-9b03-be291a9b121a"). InnerVolumeSpecName "kube-api-access-ptnq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:37.338533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.338451    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-xks-ca" pod "0d202dea-f0ad-4fe1-9b03-be291a9b121a" (UID: "0d202dea-f0ad-4fe1-9b03-be291a9b121a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:37.438693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.438657    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:37.438693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.438693    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptnq4\" (UniqueName: \"kubernetes.io/projected/0d202dea-f0ad-4fe1-9b03-be291a9b121a-kube-api-access-ptnq4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:37.438983 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:37.438720    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d202dea-f0ad-4fe1-9b03-be291a9b121a-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:38.080371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:38.080342    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"
Oct 31 07:42:38.080582 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:38.080345    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx" event={"ID":"0d202dea-f0ad-4fe1-9b03-be291a9b121a","Type":"ContainerDied","Data":"35fc2fd33024d3cc22d9500e6c41540702f51fc620986ef2e2ad39a9ea1485e9"}
Oct 31 07:42:38.080582 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:38.080464    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35fc2fd33024d3cc22d9500e6c41540702f51fc620986ef2e2ad39a9ea1485e9"
Oct 31 07:42:40.421878 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.421827    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"]
Oct 31 07:42:40.422533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.422517    5829 cpu_manager.go:414] "containerMap: removing container" podUID="0d202dea-f0ad-4fe1-9b03-be291a9b121a" containerName="revoke-keys"
Oct 31 07:42:40.422588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.422534    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d202dea-f0ad-4fe1-9b03-be291a9b121a" containerName="revoke-keys"
Oct 31 07:42:40.422674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.422664    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d202dea-f0ad-4fe1-9b03-be291a9b121a" containerName="revoke-keys"
Oct 31 07:42:40.426678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.426658    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.446442 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.446408    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"]
Oct 31 07:42:40.454263 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.454237    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:42:40.459859 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.459832    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:42:40.459859 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.459833    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:42:40.460069 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.459832    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:42:40.580085 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.580052    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2482a204-nt5lp\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.580263 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.580165    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxrq\" (UniqueName: \"kubernetes.io/projected/70bdecec-8df0-4148-aa9c-43f269ef37f5-kube-api-access-psxrq\") pod \"maas-api-revoke-sub-2482a204-nt5lp\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.580263 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.580237    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2482a204-nt5lp\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.681819 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.681728    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2482a204-nt5lp\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.682003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.681880    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxrq\" (UniqueName: \"kubernetes.io/projected/70bdecec-8df0-4148-aa9c-43f269ef37f5-kube-api-access-psxrq\") pod \"maas-api-revoke-sub-2482a204-nt5lp\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.682003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.681926    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2482a204-nt5lp\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.682621 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.682594    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2482a204-nt5lp\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.684276 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.684252    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2482a204-nt5lp\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.701573 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.701545    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxrq\" (UniqueName: \"kubernetes.io/projected/70bdecec-8df0-4148-aa9c-43f269ef37f5-kube-api-access-psxrq\") pod \"maas-api-revoke-sub-2482a204-nt5lp\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:40.740590 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:40.740558    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:41.060006 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:41.059979    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"]
Oct 31 07:42:41.061122 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:41.061095    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bdecec_8df0_4148_aa9c_43f269ef37f5.slice/crio-ab06cb254c58598cdabf4d67f44b488f17f4fd9916fd843a081cbfeb9e1aef00 WatchSource:0}: Error finding container ab06cb254c58598cdabf4d67f44b488f17f4fd9916fd843a081cbfeb9e1aef00: Status 404 returned error can't find the container with id ab06cb254c58598cdabf4d67f44b488f17f4fd9916fd843a081cbfeb9e1aef00
Oct 31 07:42:41.106799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:41.106758    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp" event={"ID":"70bdecec-8df0-4148-aa9c-43f269ef37f5","Type":"ContainerStarted","Data":"ab06cb254c58598cdabf4d67f44b488f17f4fd9916fd843a081cbfeb9e1aef00"}
Oct 31 07:42:42.116041 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:42.115956    5829 generic.go:356] "Generic (PLEG): container finished" podID="70bdecec-8df0-4148-aa9c-43f269ef37f5" containerID="73c974236c392db18c5b7293a7518127705936ab42cd9c15f8e801fb82cea524" exitCode=0
Oct 31 07:42:42.116436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:42.116044    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp" event={"ID":"70bdecec-8df0-4148-aa9c-43f269ef37f5","Type":"ContainerDied","Data":"73c974236c392db18c5b7293a7518127705936ab42cd9c15f8e801fb82cea524"}
Oct 31 07:42:43.290636 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.290609    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:43.415876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.415842    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bdecec-8df0-4148-aa9c-43f269ef37f5-kube-api-access-psxrq\" (UniqueName: \"kubernetes.io/projected/70bdecec-8df0-4148-aa9c-43f269ef37f5-kube-api-access-psxrq\") pod \"70bdecec-8df0-4148-aa9c-43f269ef37f5\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") "
Oct 31 07:42:43.416092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.415898    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-xks-ca\") pod \"70bdecec-8df0-4148-aa9c-43f269ef37f5\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") "
Oct 31 07:42:43.416092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.416051    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-openshift-ca\") pod \"70bdecec-8df0-4148-aa9c-43f269ef37f5\" (UID: \"70bdecec-8df0-4148-aa9c-43f269ef37f5\") "
Oct 31 07:42:43.416423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.416396    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-openshift-ca" pod "70bdecec-8df0-4148-aa9c-43f269ef37f5" (UID: "70bdecec-8df0-4148-aa9c-43f269ef37f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:43.418072 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.418048    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-xks-ca" pod "70bdecec-8df0-4148-aa9c-43f269ef37f5" (UID: "70bdecec-8df0-4148-aa9c-43f269ef37f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:43.418164 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.418143    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bdecec-8df0-4148-aa9c-43f269ef37f5-kube-api-access-psxrq" pod "70bdecec-8df0-4148-aa9c-43f269ef37f5" (UID: "70bdecec-8df0-4148-aa9c-43f269ef37f5"). InnerVolumeSpecName "kube-api-access-psxrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:43.517620 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.517583    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxrq\" (UniqueName: \"kubernetes.io/projected/70bdecec-8df0-4148-aa9c-43f269ef37f5-kube-api-access-psxrq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:43.517620 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.517614    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:43.517620 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:43.517625    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70bdecec-8df0-4148-aa9c-43f269ef37f5-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:44.132596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:44.132567    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"
Oct 31 07:42:44.132766 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:44.132591    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp" event={"ID":"70bdecec-8df0-4148-aa9c-43f269ef37f5","Type":"ContainerDied","Data":"ab06cb254c58598cdabf4d67f44b488f17f4fd9916fd843a081cbfeb9e1aef00"}
Oct 31 07:42:44.132766 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:44.132640    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab06cb254c58598cdabf4d67f44b488f17f4fd9916fd843a081cbfeb9e1aef00"
Oct 31 07:42:44.677221 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:44.677187    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:44.690230 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:44.690204    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:45.151884 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:45.151851    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:42:46.226322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:46.226273    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:42:46.511084 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:42:46.510998    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:42:50.111014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.110971    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"]
Oct 31 07:42:50.111714 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.111694    5829 cpu_manager.go:414] "containerMap: removing container" podUID="70bdecec-8df0-4148-aa9c-43f269ef37f5" containerName="revoke-keys"
Oct 31 07:42:50.111714 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.111715    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bdecec-8df0-4148-aa9c-43f269ef37f5" containerName="revoke-keys"
Oct 31 07:42:50.111866 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.111854    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bdecec-8df0-4148-aa9c-43f269ef37f5" containerName="revoke-keys"
Oct 31 07:42:50.116338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.116312    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.136257 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.136226    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"]
Oct 31 07:42:50.143584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.143514    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-648fbe\"/\"openshift-service-ca.crt\""
Oct 31 07:42:50.143584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.143529    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-648fbe\"/\"default-dockercfg-zgsvr\""
Oct 31 07:42:50.145295 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.145250    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-648fbe\"/\"distinct-2-w0-648fbe-kserve-self-signed-certs\""
Oct 31 07:42:50.145464 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.145253    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-648fbe\"/\"kube-root-ca.crt\""
Oct 31 07:42:50.248539 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.248506    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-tmp-dir\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.248728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.248573    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-home\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.248728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.248637    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj66f\" (UniqueName: \"kubernetes.io/projected/d4c3221f-f15b-438b-855d-379088d38324-kube-api-access-vj66f\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.248834 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.248730    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c3221f-f15b-438b-855d-379088d38324-tls-certs\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.248834 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.248795    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-dshm\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.248948 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.248923    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-model-cache\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.363569 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.363473    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-tmp-dir\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.363846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.363820    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-home\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.364069 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.364048    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj66f\" (UniqueName: \"kubernetes.io/projected/d4c3221f-f15b-438b-855d-379088d38324-kube-api-access-vj66f\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.364199 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.364184    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c3221f-f15b-438b-855d-379088d38324-tls-certs\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.364397 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.364382    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-dshm\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.364552 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.364536    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-model-cache\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.364934 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.364907    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-home\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.365025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.364209    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-tmp-dir\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.365145 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.365126    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-model-cache\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.370786 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.369902    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c3221f-f15b-438b-855d-379088d38324-tls-certs\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.370786 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.369979    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-dshm\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.390346 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.386194    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj66f\" (UniqueName: \"kubernetes.io/projected/d4c3221f-f15b-438b-855d-379088d38324-kube-api-access-vj66f\") pod \"distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") " pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.433603 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.433564    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:50.740128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.740084    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"]
Oct 31 07:42:50.741067 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:50.741031    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4c3221f_f15b_438b_855d_379088d38324.slice/crio-3de9ad8af10aa68cd10e07c6dab5353b2558a2d7fe4e290636f6e90df929f9b1 WatchSource:0}: Error finding container 3de9ad8af10aa68cd10e07c6dab5353b2558a2d7fe4e290636f6e90df929f9b1: Status 404 returned error can't find the container with id 3de9ad8af10aa68cd10e07c6dab5353b2558a2d7fe4e290636f6e90df929f9b1
Oct 31 07:42:50.936124 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.936086    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"]
Oct 31 07:42:50.940854 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.940834    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:50.959631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.959597    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"]
Oct 31 07:42:50.966907 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.966877    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:42:50.966907 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.966906    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:42:50.967149 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.966882    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:42:50.967350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:50.967327    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:42:51.073730 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.073633    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtglv\" (UniqueName: \"kubernetes.io/projected/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-kube-api-access-jtglv\") pod \"maas-api-revoke-sub-d95db595-2jc5b\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.073890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.073845    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d95db595-2jc5b\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.073930 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.073901    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d95db595-2jc5b\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.174751 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.174713    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtglv\" (UniqueName: \"kubernetes.io/projected/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-kube-api-access-jtglv\") pod \"maas-api-revoke-sub-d95db595-2jc5b\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.175128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.174861    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d95db595-2jc5b\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.175128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.174934    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d95db595-2jc5b\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.175520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.175497    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d95db595-2jc5b\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.177230 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.177211    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d95db595-2jc5b\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.195072 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.195041    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtglv\" (UniqueName: \"kubernetes.io/projected/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-kube-api-access-jtglv\") pod \"maas-api-revoke-sub-d95db595-2jc5b\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.210012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.209976    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" event={"ID":"d4c3221f-f15b-438b-855d-379088d38324","Type":"ContainerStarted","Data":"2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650"}
Oct 31 07:42:51.210012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.210018    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" event={"ID":"d4c3221f-f15b-438b-855d-379088d38324","Type":"ContainerStarted","Data":"3de9ad8af10aa68cd10e07c6dab5353b2558a2d7fe4e290636f6e90df929f9b1"}
Oct 31 07:42:51.252244 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.252186    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podStartSLOduration=1.252173193 podStartE2EDuration="1.252173193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:51.24924562 +0000 UTC m=+835.388406210" watchObservedRunningTime="2026-10-31 07:42:51.252173193 +0000 UTC m=+835.391333785"
Oct 31 07:42:51.254861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.254841    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:51.434177 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.434142    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:42:51.564485 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:42:51.564450    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dceb19a_5ea9_4b23_9d4f_163c57e36e17.slice/crio-45ac92831b567baa804af5cbdb8a485012d2e99eaa3e2abc004cb67351314837 WatchSource:0}: Error finding container 45ac92831b567baa804af5cbdb8a485012d2e99eaa3e2abc004cb67351314837: Status 404 returned error can't find the container with id 45ac92831b567baa804af5cbdb8a485012d2e99eaa3e2abc004cb67351314837
Oct 31 07:42:51.566148 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:51.566118    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"]
Oct 31 07:42:52.218613 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:52.218565    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b" event={"ID":"6dceb19a-5ea9-4b23-9d4f-163c57e36e17","Type":"ContainerStarted","Data":"bf71f314fbe5e873b6bb01ecfc770e2a754f41f261d61bde85e773c72b3ff7a8"}
Oct 31 07:42:52.219031 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:52.218635    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b" event={"ID":"6dceb19a-5ea9-4b23-9d4f-163c57e36e17","Type":"ContainerStarted","Data":"45ac92831b567baa804af5cbdb8a485012d2e99eaa3e2abc004cb67351314837"}
Oct 31 07:42:52.259759 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:52.259704    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b" podStartSLOduration=2.259688703 podStartE2EDuration="2.259688703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:52.25819957 +0000 UTC m=+836.397360158" watchObservedRunningTime="2026-10-31 07:42:52.259688703 +0000 UTC m=+836.398849290"
Oct 31 07:42:53.227055 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:53.227017    5829 generic.go:356] "Generic (PLEG): container finished" podID="6dceb19a-5ea9-4b23-9d4f-163c57e36e17" containerID="bf71f314fbe5e873b6bb01ecfc770e2a754f41f261d61bde85e773c72b3ff7a8" exitCode=0
Oct 31 07:42:53.227493 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:53.227066    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b" event={"ID":"6dceb19a-5ea9-4b23-9d4f-163c57e36e17","Type":"ContainerDied","Data":"bf71f314fbe5e873b6bb01ecfc770e2a754f41f261d61bde85e773c72b3ff7a8"}
Oct 31 07:42:54.398474 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.398447    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:54.522861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.522766    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-kube-api-access-jtglv\" (UniqueName: \"kubernetes.io/projected/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-kube-api-access-jtglv\") pod \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") "
Oct 31 07:42:54.522861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.522851    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-openshift-ca\") pod \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") "
Oct 31 07:42:54.523091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.522945    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-xks-ca\") pod \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\" (UID: \"6dceb19a-5ea9-4b23-9d4f-163c57e36e17\") "
Oct 31 07:42:54.523328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.523269    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-openshift-ca" pod "6dceb19a-5ea9-4b23-9d4f-163c57e36e17" (UID: "6dceb19a-5ea9-4b23-9d4f-163c57e36e17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:42:54.523580 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.523552    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:54.524926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.524903    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-kube-api-access-jtglv" pod "6dceb19a-5ea9-4b23-9d4f-163c57e36e17" (UID: "6dceb19a-5ea9-4b23-9d4f-163c57e36e17"). InnerVolumeSpecName "kube-api-access-jtglv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:42:54.525109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.525085    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-xks-ca" pod "6dceb19a-5ea9-4b23-9d4f-163c57e36e17" (UID: "6dceb19a-5ea9-4b23-9d4f-163c57e36e17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:42:54.624510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.624467    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtglv\" (UniqueName: \"kubernetes.io/projected/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-kube-api-access-jtglv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:54.624510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:54.624506    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dceb19a-5ea9-4b23-9d4f-163c57e36e17-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:42:55.243716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:55.243680    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b" event={"ID":"6dceb19a-5ea9-4b23-9d4f-163c57e36e17","Type":"ContainerDied","Data":"45ac92831b567baa804af5cbdb8a485012d2e99eaa3e2abc004cb67351314837"}
Oct 31 07:42:55.243716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:55.243701    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"
Oct 31 07:42:55.243931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:42:55.243742    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45ac92831b567baa804af5cbdb8a485012d2e99eaa3e2abc004cb67351314837"
Oct 31 07:43:00.434713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:00.434673    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:43:00.448139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:00.448109    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:43:01.304782 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:01.304747    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:43:02.766239 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.766197    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"]
Oct 31 07:43:02.767230 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.767207    5829 cpu_manager.go:414] "containerMap: removing container" podUID="6dceb19a-5ea9-4b23-9d4f-163c57e36e17" containerName="revoke-keys"
Oct 31 07:43:02.767329 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.767235    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dceb19a-5ea9-4b23-9d4f-163c57e36e17" containerName="revoke-keys"
Oct 31 07:43:02.767487 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.767471    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dceb19a-5ea9-4b23-9d4f-163c57e36e17" containerName="revoke-keys"
Oct 31 07:43:02.773335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.773309    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.791698 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.791670    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"]
Oct 31 07:43:02.799378 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.799354    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3cc201\"/\"distinct-2-w3-3cc201-kserve-self-signed-certs\""
Oct 31 07:43:02.836777 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.836741    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-tmp-dir\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.836951 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.836815    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cac2f4-192d-4553-87fb-a0961517fc68-tls-certs\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.836951 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.836887    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-home\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.836951 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.836925    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-dshm\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.837052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.836948    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-model-cache\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.837052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.837021    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchzc\" (UniqueName: \"kubernetes.io/projected/b2cac2f4-192d-4553-87fb-a0961517fc68-kube-api-access-fchzc\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.938425 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.938385    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cac2f4-192d-4553-87fb-a0961517fc68-tls-certs\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.939220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.939194    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-home\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.939478 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.939459    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-dshm\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.939605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.939590    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-model-cache\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.939757 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.939742    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchzc\" (UniqueName: \"kubernetes.io/projected/b2cac2f4-192d-4553-87fb-a0961517fc68-kube-api-access-fchzc\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.941051 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.940175    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-home\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.941051 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.940422    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-model-cache\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.941758 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.941714    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-tmp-dir\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.941988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.941973    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-tmp-dir\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.948656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.948590    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cac2f4-192d-4553-87fb-a0961517fc68-tls-certs\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.950580 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.950551    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-dshm\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:02.968901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:02.968863    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchzc\" (UniqueName: \"kubernetes.io/projected/b2cac2f4-192d-4553-87fb-a0961517fc68-kube-api-access-fchzc\") pod \"distinct-2-w3-3cc201-kserve-5c56557777-66lw2\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") " pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:03.086703 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.086611    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:03.397922 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.397897    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"]
Oct 31 07:43:03.399223 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:03.399197    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2cac2f4_192d_4553_87fb_a0961517fc68.slice/crio-2973159d7d00dfbf2e14f0f9949c9b20270326844d473af07ed3e373cb4d8102 WatchSource:0}: Error finding container 2973159d7d00dfbf2e14f0f9949c9b20270326844d473af07ed3e373cb4d8102: Status 404 returned error can't find the container with id 2973159d7d00dfbf2e14f0f9949c9b20270326844d473af07ed3e373cb4d8102
Oct 31 07:43:03.560725 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.560685    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"]
Oct 31 07:43:03.565273 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.565246    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.592489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.592449    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"]
Oct 31 07:43:03.654943 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.654851    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istiod-ca-cert\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.654943 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.654897    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-podinfo\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.654943 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.654920    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-token\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.655243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.655060    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.655243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.655096    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-certs\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.655243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.655126    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-data\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.655363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.655264    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-credential-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.655417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.655403    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-envoy\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.655498 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.655486    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxbj\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-kube-api-access-dlxbj\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.756536 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.756490    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-envoy\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.756739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.756590    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxbj\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-kube-api-access-dlxbj\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.756739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.756644    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istiod-ca-cert\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.756739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.756679    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-podinfo\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.756739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.756706    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-token\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.756950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.756767    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.756950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.756804    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-certs\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.756950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.756841    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-data\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.756950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.756909    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-credential-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.757189 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.757162    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.757256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.757236    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-certs\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.757341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.757265    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-data\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.757459 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.757437    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istiod-ca-cert\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.757516 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.757440    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-credential-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.758922 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.758895    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-envoy\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.759490 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.759469    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-podinfo\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.777748 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.777717    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-token\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.778109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.777907    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxbj\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-kube-api-access-dlxbj\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.798855 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.798830    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"]
Oct 31 07:43:03.804819 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.804800    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.825102 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.825077    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"]
Oct 31 07:43:03.859008 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.858969    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5752\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-kube-api-access-b5752\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.859154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.859085    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-token\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.859219 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.859205    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-data\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.859314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.859290    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-certs\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.859362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.859348    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-credential-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.859426 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.859411    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-podinfo\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.859483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.859468    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-envoy\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.859536 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.859523    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istiod-ca-cert\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.859573 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.859558    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.879345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.879317    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:03.961765 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.961714    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5752\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-kube-api-access-b5752\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.961926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.961803    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-token\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962017 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.961953    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-data\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962017 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.961987    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-certs\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962117 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.962027    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-credential-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962117 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.962091    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-podinfo\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.962132    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-envoy\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.962158    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istiod-ca-cert\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.962199    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962505 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.962480    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-data\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962608 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.962532    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-certs\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.962682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.962655    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.963201 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.963177    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-credential-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.963350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.963262    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istiod-ca-cert\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.965143 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.965118    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-envoy\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.965537 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.965519    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-podinfo\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.984404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.984368    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5752\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-kube-api-access-b5752\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:03.985406 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:03.985378    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-token\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:04.119838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.119802    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:04.219526 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.219485    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:04.219654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.219580    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:04.219654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.219628    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:04.227339 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.227309    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"]
Oct 31 07:43:04.320001 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.319961    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" event={"ID":"6925ac9a-0be2-4520-96fa-f4126b2fe58b","Type":"ContainerStarted","Data":"8b70f4cd38f42e16e01263a4dbed47b785dbfdc4dd22acc141cef9beb3d5a840"}
Oct 31 07:43:04.320192 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.320041    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" event={"ID":"6925ac9a-0be2-4520-96fa-f4126b2fe58b","Type":"ContainerStarted","Data":"595ca5462afb24705b040cde51f82584e382c25a3db9c7d3833794c76be5604a"}
Oct 31 07:43:04.322428 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.322399    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" event={"ID":"b2cac2f4-192d-4553-87fb-a0961517fc68","Type":"ContainerStarted","Data":"d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750"}
Oct 31 07:43:04.322553 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.322441    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" event={"ID":"b2cac2f4-192d-4553-87fb-a0961517fc68","Type":"ContainerStarted","Data":"2973159d7d00dfbf2e14f0f9949c9b20270326844d473af07ed3e373cb4d8102"}
Oct 31 07:43:04.367151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.367085    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podStartSLOduration=1.367066634 podStartE2EDuration="1.367066634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:04.365994117 +0000 UTC m=+848.505154709" watchObservedRunningTime="2026-10-31 07:43:04.367066634 +0000 UTC m=+848.506227224"
Oct 31 07:43:04.410852 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.410801    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podStartSLOduration=2.410786649 podStartE2EDuration="2.410786649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:04.409502073 +0000 UTC m=+848.548662661" watchObservedRunningTime="2026-10-31 07:43:04.410786649 +0000 UTC m=+848.549947236"
Oct 31 07:43:04.430525 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:04.430482    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a46c7a_6f14_42d9_9012_f4177e2fbad8.slice/crio-efd43133e3ff5551c8068e49f9e604380f19d3ae4b684ad576a61acc400119f1 WatchSource:0}: Error finding container efd43133e3ff5551c8068e49f9e604380f19d3ae4b684ad576a61acc400119f1: Status 404 returned error can't find the container with id efd43133e3ff5551c8068e49f9e604380f19d3ae4b684ad576a61acc400119f1
Oct 31 07:43:04.430525 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.430494    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"]
Oct 31 07:43:04.433074 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.433040    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:04.433151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.433113    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:04.433151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.433143    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:04.880017 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:04.879932    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:05.087600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:05.087573    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:05.368463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:05.368416    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" event={"ID":"05a46c7a-6f14-42d9-9012-f4177e2fbad8","Type":"ContainerStarted","Data":"7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff"}
Oct 31 07:43:05.368748 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:05.368718    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" event={"ID":"05a46c7a-6f14-42d9-9012-f4177e2fbad8","Type":"ContainerStarted","Data":"efd43133e3ff5551c8068e49f9e604380f19d3ae4b684ad576a61acc400119f1"}
Oct 31 07:43:05.881436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:05.881395    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 07:43:05.881831 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:05.881450    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 07:43:06.125151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:06.125118    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:06.125583 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:06.125554    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 07:43:06.125830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:06.125804    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 07:43:06.880235 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:06.880201    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 07:43:06.880412 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:06.880252    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 07:43:07.128798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:07.128757    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 07:43:07.129457 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:07.129423    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 07:43:07.879744 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:07.879702    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 07:43:07.879927 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:07.879765    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 07:43:08.128309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:08.121702    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 07:43:08.128309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:08.121762    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 07:43:08.880117 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:08.880082    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 07:43:08.880627 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:08.880170    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 07:43:09.122187 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:09.121723    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 07:43:09.122187 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:09.121799    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 07:43:09.880112 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:09.880070    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 07:43:09.880319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:09.880138    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 07:43:10.122237 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:10.121741    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 07:43:10.122237 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:10.121810    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 07:43:10.879747 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:10.879707    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Oct 31 07:43:10.879944 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:10.879774    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Oct 31 07:43:11.120556 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:11.120516    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 31 07:43:11.121099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:11.120578    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 31 07:43:11.885560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:11.885515    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:11.886095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:11.886072    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:11.886938 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:11.886910    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:43:11.947271 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:11.947217    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" podStartSLOduration=8.947202507 podStartE2EDuration="8.947202507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:05.458507898 +0000 UTC m=+849.597668500" watchObservedRunningTime="2026-10-31 07:43:11.947202507 +0000 UTC m=+856.086363095"
Oct 31 07:43:12.132562 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:12.132514    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:12.133100 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:12.133065    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:12.134090 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:12.134053    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:43:13.087217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:13.087178    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:13.100111 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:13.100086    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:13.486753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:13.486724    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:16.510808 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:43:16.510769    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:43:18.225854 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.225819    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:43:18.730143 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.730104    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"]
Oct 31 07:43:18.735035 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.735008    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.764962 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.764928    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"]
Oct 31 07:43:18.836256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.826673    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-certs\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.836256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.835466    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-data\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.836256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.835728    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.836256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.835757    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-podinfo\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.836256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.835811    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-envoy\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.836256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.835841    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-token\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.836256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.835980    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-credential-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.836256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.836003    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istiod-ca-cert\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.836256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.836034    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f7tn\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-kube-api-access-6f7tn\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.967801 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.967762    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-envoy\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.967801 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.967804    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-token\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.967885    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-credential-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968049    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istiod-ca-cert\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968209 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968102    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f7tn\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-kube-api-access-6f7tn\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968209 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968160    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-certs\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968209 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968193    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-data\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968403 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968273    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-credential-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968425    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968454    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-podinfo\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968585 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968539    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-certs\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968585 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968559    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-data\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968778 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968758    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.968854 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.968785    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istiod-ca-cert\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.970239 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.970212    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-envoy\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.970806 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.970784    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-podinfo\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.994800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.994716    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-token\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:18.994800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:18.994742    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f7tn\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-kube-api-access-6f7tn\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:19.051366 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:19.051329    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:19.384415 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:19.384383    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"]
Oct 31 07:43:19.384822 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:19.384772    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a750593_3ebd_4a88_b317_8ccd6aaddb8f.slice/crio-1bb7b9017ef3a16b0c29bc04810816466437e81e534a8c25c3e8c8c6a48400f1 WatchSource:0}: Error finding container 1bb7b9017ef3a16b0c29bc04810816466437e81e534a8c25c3e8c8c6a48400f1: Status 404 returned error can't find the container with id 1bb7b9017ef3a16b0c29bc04810816466437e81e534a8c25c3e8c8c6a48400f1
Oct 31 07:43:19.387495 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:19.387459    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:19.387561 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:19.387532    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:19.387600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:19.387560    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:19.536952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:19.535076    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" event={"ID":"0a750593-3ebd-4a88-b317-8ccd6aaddb8f","Type":"ContainerStarted","Data":"f7e16fb1d7b4a1c83af4015348128efa12b36e4777411ac7674c5d03081a1d42"}
Oct 31 07:43:19.536952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:19.535146    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" event={"ID":"0a750593-3ebd-4a88-b317-8ccd6aaddb8f","Type":"ContainerStarted","Data":"1bb7b9017ef3a16b0c29bc04810816466437e81e534a8c25c3e8c8c6a48400f1"}
Oct 31 07:43:19.609108 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:19.604733    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" podStartSLOduration=1.604709457 podStartE2EDuration="1.604709457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:19.588747146 +0000 UTC m=+863.727907759" watchObservedRunningTime="2026-10-31 07:43:19.604709457 +0000 UTC m=+863.743870046"
Oct 31 07:43:20.052454 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:20.052225    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:20.053800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:20.053773    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 07:43:20.053947 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:20.053841    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 07:43:21.052521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:21.052479    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 07:43:21.053022 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:21.052543    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 07:43:22.052801 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:22.052762    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 07:43:22.053214 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:22.052820    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 07:43:23.052506 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:23.052466    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 07:43:23.052692 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:23.052538    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 07:43:24.051799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.051758    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 07:43:24.052229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.051827    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 07:43:24.307382 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.304683    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"]
Oct 31 07:43:24.335315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.334987    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.339354 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.339322    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"]
Oct 31 07:43:24.352464 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.352434    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:43:24.352589 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.352575    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:43:24.352683 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.352668    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:43:24.365460 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.365426    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:43:24.376244 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.376215    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d700421-47mdl\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.376435 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.376301    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d700421-47mdl\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.376609 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.376532    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbrdj\" (UniqueName: \"kubernetes.io/projected/5502cd79-b262-407f-8a2f-2517aa1e7cb7-kube-api-access-pbrdj\") pod \"maas-api-revoke-sub-1d700421-47mdl\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.478134 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.478101    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d700421-47mdl\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.478451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.478176    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d700421-47mdl\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.478451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.478271    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbrdj\" (UniqueName: \"kubernetes.io/projected/5502cd79-b262-407f-8a2f-2517aa1e7cb7-kube-api-access-pbrdj\") pod \"maas-api-revoke-sub-1d700421-47mdl\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.478916 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.478892    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d700421-47mdl\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.480786 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.480761    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d700421-47mdl\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.500897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.500866    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbrdj\" (UniqueName: \"kubernetes.io/projected/5502cd79-b262-407f-8a2f-2517aa1e7cb7-kube-api-access-pbrdj\") pod \"maas-api-revoke-sub-1d700421-47mdl\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.654465 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.654433    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:24.985749 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:24.985721    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"]
Oct 31 07:43:24.986673 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:24.986644    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5502cd79_b262_407f_8a2f_2517aa1e7cb7.slice/crio-b3dfa31be2554402eab9ca6a225c51f2dd2f62a20320e597269a0b395d97beda WatchSource:0}: Error finding container b3dfa31be2554402eab9ca6a225c51f2dd2f62a20320e597269a0b395d97beda: Status 404 returned error can't find the container with id b3dfa31be2554402eab9ca6a225c51f2dd2f62a20320e597269a0b395d97beda
Oct 31 07:43:25.052374 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:25.052332    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 07:43:25.052695 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:25.052398    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 07:43:25.624400 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:25.624302    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl" event={"ID":"5502cd79-b262-407f-8a2f-2517aa1e7cb7","Type":"ContainerStarted","Data":"9c8c42737f9abeef72cc5cfd0d2ce07d7d41a09f2d5aec0584a4a2488792017d"}
Oct 31 07:43:25.624400 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:25.624372    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl" event={"ID":"5502cd79-b262-407f-8a2f-2517aa1e7cb7","Type":"ContainerStarted","Data":"b3dfa31be2554402eab9ca6a225c51f2dd2f62a20320e597269a0b395d97beda"}
Oct 31 07:43:25.664667 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:25.664590    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl" podStartSLOduration=1.664568236 podStartE2EDuration="1.664568236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:25.662787868 +0000 UTC m=+869.801948458" watchObservedRunningTime="2026-10-31 07:43:25.664568236 +0000 UTC m=+869.803728823"
Oct 31 07:43:26.051925 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:26.051883    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 31 07:43:26.052135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:26.051967    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 31 07:43:26.634875 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:26.634836    5829 generic.go:356] "Generic (PLEG): container finished" podID="5502cd79-b262-407f-8a2f-2517aa1e7cb7" containerID="9c8c42737f9abeef72cc5cfd0d2ce07d7d41a09f2d5aec0584a4a2488792017d" exitCode=0
Oct 31 07:43:26.635455 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:26.634878    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl" event={"ID":"5502cd79-b262-407f-8a2f-2517aa1e7cb7","Type":"ContainerDied","Data":"9c8c42737f9abeef72cc5cfd0d2ce07d7d41a09f2d5aec0584a4a2488792017d"}
Oct 31 07:43:27.056825 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.056794    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:27.057315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.057270    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:27.058107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.058088    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:43:27.812781 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.812741    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:27.935704 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.935613    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-openshift-ca\") pod \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") "
Oct 31 07:43:27.935876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.935728    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-xks-ca\") pod \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") "
Oct 31 07:43:27.935940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.935889    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5502cd79-b262-407f-8a2f-2517aa1e7cb7-kube-api-access-pbrdj\" (UniqueName: \"kubernetes.io/projected/5502cd79-b262-407f-8a2f-2517aa1e7cb7-kube-api-access-pbrdj\") pod \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\" (UID: \"5502cd79-b262-407f-8a2f-2517aa1e7cb7\") "
Oct 31 07:43:27.936000 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.935955    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-openshift-ca" pod "5502cd79-b262-407f-8a2f-2517aa1e7cb7" (UID: "5502cd79-b262-407f-8a2f-2517aa1e7cb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:43:27.936367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.936346    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:27.937870 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.937849    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5502cd79-b262-407f-8a2f-2517aa1e7cb7-kube-api-access-pbrdj" pod "5502cd79-b262-407f-8a2f-2517aa1e7cb7" (UID: "5502cd79-b262-407f-8a2f-2517aa1e7cb7"). InnerVolumeSpecName "kube-api-access-pbrdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:43:27.937943 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:27.937871    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-xks-ca" pod "5502cd79-b262-407f-8a2f-2517aa1e7cb7" (UID: "5502cd79-b262-407f-8a2f-2517aa1e7cb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:43:28.038198 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:28.038155    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbrdj\" (UniqueName: \"kubernetes.io/projected/5502cd79-b262-407f-8a2f-2517aa1e7cb7-kube-api-access-pbrdj\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:28.038198 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:28.038192    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5502cd79-b262-407f-8a2f-2517aa1e7cb7-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:28.653084 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:28.653053    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"
Oct 31 07:43:28.653266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:28.653083    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl" event={"ID":"5502cd79-b262-407f-8a2f-2517aa1e7cb7","Type":"ContainerDied","Data":"b3dfa31be2554402eab9ca6a225c51f2dd2f62a20320e597269a0b395d97beda"}
Oct 31 07:43:28.653266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:28.653136    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3dfa31be2554402eab9ca6a225c51f2dd2f62a20320e597269a0b395d97beda"
Oct 31 07:43:37.483753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.483714    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"]
Oct 31 07:43:37.484494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.484469    5829 cpu_manager.go:414] "containerMap: removing container" podUID="5502cd79-b262-407f-8a2f-2517aa1e7cb7" containerName="revoke-keys"
Oct 31 07:43:37.484494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.484490    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5502cd79-b262-407f-8a2f-2517aa1e7cb7" containerName="revoke-keys"
Oct 31 07:43:37.484681 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.484661    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="5502cd79-b262-407f-8a2f-2517aa1e7cb7" containerName="revoke-keys"
Oct 31 07:43:37.489716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.489694    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.505857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.505820    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"]
Oct 31 07:43:37.512217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.512183    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-027d2a\"/\"openshift-service-ca.crt\""
Oct 31 07:43:37.512415 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.512373    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-027d2a\"/\"kube-root-ca.crt\""
Oct 31 07:43:37.517723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.517700    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-027d2a\"/\"default-dockercfg-jktgf\""
Oct 31 07:43:37.517723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.517711    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-027d2a\"/\"distinct-2-w6-027d2a-kserve-self-signed-certs\""
Oct 31 07:43:37.650547 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.650504    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"]
Oct 31 07:43:37.655359 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.655329    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:37.671581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.671546    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:43:37.671807 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.671781    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:43:37.672203 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.671832    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:43:37.672519 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.672472    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:43:37.673979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.673860    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-home\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.684104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.680194    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-dshm\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.684104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.680344    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-model-cache\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.684104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.680387    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-tmp-dir\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.684104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.680446    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a158b083-77df-454a-af0d-e919c4b320c0-tls-certs\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.684104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.680488    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jslbz\" (UniqueName: \"kubernetes.io/projected/a158b083-77df-454a-af0d-e919c4b320c0-kube-api-access-jslbz\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.739033 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.738946    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"]
Oct 31 07:43:37.782756 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.782716    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnn4q\" (UniqueName: \"kubernetes.io/projected/aff220bf-a427-4dbe-b053-bc59908a2a3e-kube-api-access-rnn4q\") pod \"maas-api-revoke-sub-d48db09e-qqnwf\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:37.782953 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.782765    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-model-cache\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.782953 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.782794    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-tmp-dir\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.782953 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.782900    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a158b083-77df-454a-af0d-e919c4b320c0-tls-certs\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.782953 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.782949    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jslbz\" (UniqueName: \"kubernetes.io/projected/a158b083-77df-454a-af0d-e919c4b320c0-kube-api-access-jslbz\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.783174 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.783007    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48db09e-qqnwf\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:37.783233 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.783178    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48db09e-qqnwf\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:37.783233 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.783215    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-tmp-dir\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.783375 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.783270    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-model-cache\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.783600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.783581    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-home\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.783682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.783644    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-dshm\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.784224 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.784197    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-home\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.786027 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.786002    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-dshm\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.786520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.786503    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a158b083-77df-454a-af0d-e919c4b320c0-tls-certs\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.802697 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.802669    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jslbz\" (UniqueName: \"kubernetes.io/projected/a158b083-77df-454a-af0d-e919c4b320c0-kube-api-access-jslbz\") pod \"distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") " pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.809575 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.809554    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:37.894931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.885717    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnn4q\" (UniqueName: \"kubernetes.io/projected/aff220bf-a427-4dbe-b053-bc59908a2a3e-kube-api-access-rnn4q\") pod \"maas-api-revoke-sub-d48db09e-qqnwf\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:37.894931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.885826    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48db09e-qqnwf\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:37.894931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.885991    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48db09e-qqnwf\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:37.894931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.887151    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d48db09e-qqnwf\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:37.901521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.900651    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d48db09e-qqnwf\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:37.906977 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:37.906887    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnn4q\" (UniqueName: \"kubernetes.io/projected/aff220bf-a427-4dbe-b053-bc59908a2a3e-kube-api-access-rnn4q\") pod \"maas-api-revoke-sub-d48db09e-qqnwf\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:38.035344 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.035219    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:38.128992 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.128948    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"]
Oct 31 07:43:38.133718 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:38.133683    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda158b083_77df_454a_af0d_e919c4b320c0.slice/crio-15586bf617bb7766732dbcc4e7fe9718d04b48459459e7f51b58f82a62828f62 WatchSource:0}: Error finding container 15586bf617bb7766732dbcc4e7fe9718d04b48459459e7f51b58f82a62828f62: Status 404 returned error can't find the container with id 15586bf617bb7766732dbcc4e7fe9718d04b48459459e7f51b58f82a62828f62
Oct 31 07:43:38.343208 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.343177    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"]
Oct 31 07:43:38.343968 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:38.343940    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff220bf_a427_4dbe_b053_bc59908a2a3e.slice/crio-18835fe0db21fec9224dcd13f9dd351d14b378358b12c62285281a5ad4f067ff WatchSource:0}: Error finding container 18835fe0db21fec9224dcd13f9dd351d14b378358b12c62285281a5ad4f067ff: Status 404 returned error can't find the container with id 18835fe0db21fec9224dcd13f9dd351d14b378358b12c62285281a5ad4f067ff
Oct 31 07:43:38.441115 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.441080    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"]
Oct 31 07:43:38.447128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.447106    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.464478 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.464449    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"]
Oct 31 07:43:38.497153 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.497112    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv9fp\" (UniqueName: \"kubernetes.io/projected/54b2f342-3d23-42f2-a068-e42f57cf0366-kube-api-access-xv9fp\") pod \"maas-api-revoke-sub-976fb0cc-sc5lm\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.497674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.497256    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976fb0cc-sc5lm\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.497674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.497395    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976fb0cc-sc5lm\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.599119 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.598998    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976fb0cc-sc5lm\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.600417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.599917    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv9fp\" (UniqueName: \"kubernetes.io/projected/54b2f342-3d23-42f2-a068-e42f57cf0366-kube-api-access-xv9fp\") pod \"maas-api-revoke-sub-976fb0cc-sc5lm\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.600581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.600519    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976fb0cc-sc5lm\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.601478 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.601443    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-976fb0cc-sc5lm\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.606570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.606540    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-xks-ca\") pod \"maas-api-revoke-sub-976fb0cc-sc5lm\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.620923 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.620877    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv9fp\" (UniqueName: \"kubernetes.io/projected/54b2f342-3d23-42f2-a068-e42f57cf0366-kube-api-access-xv9fp\") pod \"maas-api-revoke-sub-976fb0cc-sc5lm\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.740337 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.740302    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf" event={"ID":"aff220bf-a427-4dbe-b053-bc59908a2a3e","Type":"ContainerStarted","Data":"1efb939e9cb75b342959711fa55dab170332c11a4e08eabe86f7e65aeb66a979"}
Oct 31 07:43:38.740507 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.740366    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf" event={"ID":"aff220bf-a427-4dbe-b053-bc59908a2a3e","Type":"ContainerStarted","Data":"18835fe0db21fec9224dcd13f9dd351d14b378358b12c62285281a5ad4f067ff"}
Oct 31 07:43:38.742084 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.742061    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" event={"ID":"a158b083-77df-454a-af0d-e919c4b320c0","Type":"ContainerStarted","Data":"819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846"}
Oct 31 07:43:38.742182 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.742092    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" event={"ID":"a158b083-77df-454a-af0d-e919c4b320c0","Type":"ContainerStarted","Data":"15586bf617bb7766732dbcc4e7fe9718d04b48459459e7f51b58f82a62828f62"}
Oct 31 07:43:38.761316 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.761292    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:38.781936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.781889    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf" podStartSLOduration=1.7818751430000002 podStartE2EDuration="1.781875143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:38.77821432 +0000 UTC m=+882.917374908" watchObservedRunningTime="2026-10-31 07:43:38.781875143 +0000 UTC m=+882.921035731"
Oct 31 07:43:38.810253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.810220    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:38.818919 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:38.818858    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podStartSLOduration=1.818839477 podStartE2EDuration="1.818839477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:38.817450597 +0000 UTC m=+882.956611188" watchObservedRunningTime="2026-10-31 07:43:38.818839477 +0000 UTC m=+882.958000065"
Oct 31 07:43:39.071967 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:39.071923    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"]
Oct 31 07:43:39.072213 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:39.072190    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b2f342_3d23_42f2_a068_e42f57cf0366.slice/crio-ba68ea999b324adb55d7b761a012ca705a0ab172c27285165da799ec4515f631 WatchSource:0}: Error finding container ba68ea999b324adb55d7b761a012ca705a0ab172c27285165da799ec4515f631: Status 404 returned error can't find the container with id ba68ea999b324adb55d7b761a012ca705a0ab172c27285165da799ec4515f631
Oct 31 07:43:39.750921 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:39.750888    5829 generic.go:356] "Generic (PLEG): container finished" podID="aff220bf-a427-4dbe-b053-bc59908a2a3e" containerID="1efb939e9cb75b342959711fa55dab170332c11a4e08eabe86f7e65aeb66a979" exitCode=0
Oct 31 07:43:39.751376 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:39.750971    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf" event={"ID":"aff220bf-a427-4dbe-b053-bc59908a2a3e","Type":"ContainerDied","Data":"1efb939e9cb75b342959711fa55dab170332c11a4e08eabe86f7e65aeb66a979"}
Oct 31 07:43:39.752826 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:39.752800    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm" event={"ID":"54b2f342-3d23-42f2-a068-e42f57cf0366","Type":"ContainerStarted","Data":"e074f211f3d6303511151e82545f135b02224f91a1fae826369f645cb3c6978d"}
Oct 31 07:43:39.752993 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:39.752851    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm" event={"ID":"54b2f342-3d23-42f2-a068-e42f57cf0366","Type":"ContainerStarted","Data":"ba68ea999b324adb55d7b761a012ca705a0ab172c27285165da799ec4515f631"}
Oct 31 07:43:39.832143 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:39.832085    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm" podStartSLOduration=1.8320696440000002 podStartE2EDuration="1.832069644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:39.827589114 +0000 UTC m=+883.966749703" watchObservedRunningTime="2026-10-31 07:43:39.832069644 +0000 UTC m=+883.971230250"
Oct 31 07:43:40.762275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:40.762237    5829 generic.go:356] "Generic (PLEG): container finished" podID="54b2f342-3d23-42f2-a068-e42f57cf0366" containerID="e074f211f3d6303511151e82545f135b02224f91a1fae826369f645cb3c6978d" exitCode=0
Oct 31 07:43:40.762753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:40.762321    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm" event={"ID":"54b2f342-3d23-42f2-a068-e42f57cf0366","Type":"ContainerDied","Data":"e074f211f3d6303511151e82545f135b02224f91a1fae826369f645cb3c6978d"}
Oct 31 07:43:40.939708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:40.939677    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:41.035024 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.034929    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-xks-ca\") pod \"aff220bf-a427-4dbe-b053-bc59908a2a3e\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") "
Oct 31 07:43:41.035183 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.035092    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff220bf-a427-4dbe-b053-bc59908a2a3e-kube-api-access-rnn4q\" (UniqueName: \"kubernetes.io/projected/aff220bf-a427-4dbe-b053-bc59908a2a3e-kube-api-access-rnn4q\") pod \"aff220bf-a427-4dbe-b053-bc59908a2a3e\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") "
Oct 31 07:43:41.035183 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.035125    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-openshift-ca\") pod \"aff220bf-a427-4dbe-b053-bc59908a2a3e\" (UID: \"aff220bf-a427-4dbe-b053-bc59908a2a3e\") "
Oct 31 07:43:41.035564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.035529    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-openshift-ca" pod "aff220bf-a427-4dbe-b053-bc59908a2a3e" (UID: "aff220bf-a427-4dbe-b053-bc59908a2a3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:43:41.035867 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.035847    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:41.037216 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.037194    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff220bf-a427-4dbe-b053-bc59908a2a3e-kube-api-access-rnn4q" pod "aff220bf-a427-4dbe-b053-bc59908a2a3e" (UID: "aff220bf-a427-4dbe-b053-bc59908a2a3e"). InnerVolumeSpecName "kube-api-access-rnn4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:43:41.037368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.037232    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-xks-ca" pod "aff220bf-a427-4dbe-b053-bc59908a2a3e" (UID: "aff220bf-a427-4dbe-b053-bc59908a2a3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:43:41.137711 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.137672    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnn4q\" (UniqueName: \"kubernetes.io/projected/aff220bf-a427-4dbe-b053-bc59908a2a3e-kube-api-access-rnn4q\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:41.137711 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.137705    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff220bf-a427-4dbe-b053-bc59908a2a3e-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:41.771683 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.771658    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"
Oct 31 07:43:41.772107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.771651    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf" event={"ID":"aff220bf-a427-4dbe-b053-bc59908a2a3e","Type":"ContainerDied","Data":"18835fe0db21fec9224dcd13f9dd351d14b378358b12c62285281a5ad4f067ff"}
Oct 31 07:43:41.772107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.771785    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18835fe0db21fec9224dcd13f9dd351d14b378358b12c62285281a5ad4f067ff"
Oct 31 07:43:41.947901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:41.947875    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:42.048765 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.048688    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b2f342-3d23-42f2-a068-e42f57cf0366-kube-api-access-xv9fp\" (UniqueName: \"kubernetes.io/projected/54b2f342-3d23-42f2-a068-e42f57cf0366-kube-api-access-xv9fp\") pod \"54b2f342-3d23-42f2-a068-e42f57cf0366\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") "
Oct 31 07:43:42.048921 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.048787    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-openshift-ca\") pod \"54b2f342-3d23-42f2-a068-e42f57cf0366\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") "
Oct 31 07:43:42.048921 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.048809    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-xks-ca\") pod \"54b2f342-3d23-42f2-a068-e42f57cf0366\" (UID: \"54b2f342-3d23-42f2-a068-e42f57cf0366\") "
Oct 31 07:43:42.049213 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.049185    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-openshift-ca" pod "54b2f342-3d23-42f2-a068-e42f57cf0366" (UID: "54b2f342-3d23-42f2-a068-e42f57cf0366"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:43:42.049462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.049437    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:42.050880 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.050856    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-xks-ca" pod "54b2f342-3d23-42f2-a068-e42f57cf0366" (UID: "54b2f342-3d23-42f2-a068-e42f57cf0366"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:43:42.050951 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.050876    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b2f342-3d23-42f2-a068-e42f57cf0366-kube-api-access-xv9fp" pod "54b2f342-3d23-42f2-a068-e42f57cf0366" (UID: "54b2f342-3d23-42f2-a068-e42f57cf0366"). InnerVolumeSpecName "kube-api-access-xv9fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:43:42.150295 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.150255    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv9fp\" (UniqueName: \"kubernetes.io/projected/54b2f342-3d23-42f2-a068-e42f57cf0366-kube-api-access-xv9fp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:42.150468 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.150304    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54b2f342-3d23-42f2-a068-e42f57cf0366-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:42.781235 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.781202    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm" event={"ID":"54b2f342-3d23-42f2-a068-e42f57cf0366","Type":"ContainerDied","Data":"ba68ea999b324adb55d7b761a012ca705a0ab172c27285165da799ec4515f631"}
Oct 31 07:43:42.781235 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.781222    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"
Oct 31 07:43:42.781675 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:42.781252    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba68ea999b324adb55d7b761a012ca705a0ab172c27285165da799ec4515f631"
Oct 31 07:43:46.710017 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:43:46.709974    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:43:47.145444 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.145386    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:47.199417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.199374    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:47.299422 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.299377    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 07:43:47.310270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.309370    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:47.430368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.429627    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:47.455390 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.452379    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:47.475710 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.474813    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:47.501307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.496377    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:47.573442 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.573391    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:47.816362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:47.815807    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:48.203418 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.203367    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 07:43:48.204357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.204332    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:43:48.302315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.286669    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:48.377304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.329409    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:48.377304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.330188    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:43:48.377688 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.377648    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 07:43:48.438038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.437974    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:48.439036 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.439016    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:43:48.485355 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.477810    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:43:48.499327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.499224    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:48.508184 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.508156    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:43:48.535347 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.535320    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:43:48.545332 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.544345    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"]
Oct 31 07:43:48.545530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.545497    5829 cpu_manager.go:414] "containerMap: removing container" podUID="aff220bf-a427-4dbe-b053-bc59908a2a3e" containerName="revoke-keys"
Oct 31 07:43:48.545530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.545521    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff220bf-a427-4dbe-b053-bc59908a2a3e" containerName="revoke-keys"
Oct 31 07:43:48.545650 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.545542    5829 cpu_manager.go:414] "containerMap: removing container" podUID="54b2f342-3d23-42f2-a068-e42f57cf0366" containerName="revoke-keys"
Oct 31 07:43:48.545650 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.545550    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b2f342-3d23-42f2-a068-e42f57cf0366" containerName="revoke-keys"
Oct 31 07:43:48.545755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.545728    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff220bf-a427-4dbe-b053-bc59908a2a3e" containerName="revoke-keys"
Oct 31 07:43:48.545755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.545744    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b2f342-3d23-42f2-a068-e42f57cf0366" containerName="revoke-keys"
Oct 31 07:43:48.677343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.676579    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:48.677343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.676615    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"]
Oct 31 07:43:48.678683 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.677958    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.716685 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.716647    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.719581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.719548    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-certs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.719812 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.719790    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-data\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.720073 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.720051    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-podinfo\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.720477 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.720391    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07174d73-53ba-408f-bf2f-6f4273be8fe3-istiod-ca-cert\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.720780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.720686    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-credential-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.721066 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.721045    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-envoy\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.721298 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.721183    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-token\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.721501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.721482    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2m8q\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-kube-api-access-j2m8q\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.737015 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.736928    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"]
Oct 31 07:43:48.737312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.737274    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"]
Oct 31 07:43:48.750358 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.750333    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:43:48.754926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.754777    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:43:48.755168 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.755148    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"]
Oct 31 07:43:48.765723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.765700    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.796649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.796600    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.803375 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.802679    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"]
Oct 31 07:43:48.853798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.853756    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a57dee9-c39b-4039-b990-889f7ecb0d96-istiod-ca-cert\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.854169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.853835    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqs92\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-kube-api-access-nqs92\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.854169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.853886    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-certs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.854169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.853951    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-certs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.854169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.853986    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-data\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.854169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854024    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-podinfo\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.854169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854053    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07174d73-53ba-408f-bf2f-6f4273be8fe3-istiod-ca-cert\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.854169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854100    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.854169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854135    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-credential-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.854438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854176    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt9jj\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-kube-api-access-mt9jj\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.854438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854226    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-credential-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.854438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854306    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-istio-token\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.854438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854357    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-data\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.854438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854390    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-envoy\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.854438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854418    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-token\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.854626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854462    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-envoy\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.854626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854506    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2m8q\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-kube-api-access-j2m8q\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.854626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854551    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-certs\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.854626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854597    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.854738 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854625    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-podinfo\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.854738 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854663    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-data\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.854738 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854693    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-credential-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.854826 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854744    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-token\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.854826 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854773    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b86fad7-2774-4996-9022-532c72f7e325-istio-podinfo\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.854826 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854808    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b86fad7-2774-4996-9022-532c72f7e325-istiod-ca-cert\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.854913 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854885    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-envoy\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.854943 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.854930    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.855494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.855463    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.858130 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.857661    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-certs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.858130 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.857834    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-data\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.859559 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.859332    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-credential-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.860791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.860767    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07174d73-53ba-408f-bf2f-6f4273be8fe3-istiod-ca-cert\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.861649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.861603    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-podinfo\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.862602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.862578    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-envoy\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.880483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.880349    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-token\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.883635 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.883488    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2m8q\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-kube-api-access-j2m8q\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:48.956167 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956131    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-istio-token\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.956377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956193    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-data\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.956377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956239    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-envoy\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.956377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956304    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-certs\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.956377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956342    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.956377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956366    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-podinfo\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.956664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956396    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-data\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.956664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956446    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-credential-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.956664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956505    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-token\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.956664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956536    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b86fad7-2774-4996-9022-532c72f7e325-istio-podinfo\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.956664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956566    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b86fad7-2774-4996-9022-532c72f7e325-istiod-ca-cert\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.956664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956646    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-envoy\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.956957 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956742    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a57dee9-c39b-4039-b990-889f7ecb0d96-istiod-ca-cert\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.956957 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956761    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.956957 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956761    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-data\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.956957 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956796    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-certs\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.956957 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.956832    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-data\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.957211 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.957095    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-credential-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.957391 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.957368    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqs92\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-kube-api-access-nqs92\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.957599 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.957571    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b86fad7-2774-4996-9022-532c72f7e325-istiod-ca-cert\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.957678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.957605    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-certs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.957678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.957631    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a57dee9-c39b-4039-b990-889f7ecb0d96-istiod-ca-cert\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.957862 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.957839    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.957950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.957933    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt9jj\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-kube-api-access-mt9jj\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.958012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.957943    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-certs\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.958073 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.958029    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-credential-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.958458 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.958430    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-credential-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.958765 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.958738    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-socket\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.959191 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.959163    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-envoy\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.959774 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.959748    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-envoy\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.959918 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.959901    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-podinfo\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.959979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.959941    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b86fad7-2774-4996-9022-532c72f7e325-istio-podinfo\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.976044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.976020    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-istio-token\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:48.982812 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.982779    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqs92\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-kube-api-access-nqs92\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.983691 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.983648    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-token\") pod \"e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") " pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:48.984245 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:48.984221    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt9jj\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-kube-api-access-mt9jj\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:49.086602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.086511    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:49.123706 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.123440    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:49.130979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.130940    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:49.159421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.159388    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"]
Oct 31 07:43:49.172566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.170990    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.200578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.199254    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"]
Oct 31 07:43:49.265417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.265375    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-certs\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.265417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.265462    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-credential-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.265417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.265531    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-envoy\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.265417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.265615    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istiod-ca-cert\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.265417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.265716    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cscnl\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-kube-api-access-cscnl\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.266151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.265779    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-token\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.266151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.266123    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-podinfo\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.266251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.266184    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-data\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.266407 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.266329    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.368551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.368461    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istiod-ca-cert\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.368681 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.368574    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cscnl\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-kube-api-access-cscnl\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.368681 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.368618    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-token\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.368762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.368710    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-podinfo\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.368762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.368729    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-data\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.368820 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.368762    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.368820 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.368797    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-certs\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.368881 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.368834    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-credential-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.368881 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.368876    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-envoy\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.369606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.369306    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istiod-ca-cert\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.369606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.369381    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-certs\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.369606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.369406    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.369606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.369493    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-data\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.369606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.369588    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-credential-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.371439 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.371416    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-envoy\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.371653 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.371637    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-podinfo\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.392202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.392173    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-token\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.392349 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.392259    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cscnl\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-kube-api-access-cscnl\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.430269 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.430243    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"]
Oct 31 07:43:49.432452 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:49.432423    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07174d73_53ba_408f_bf2f_6f4273be8fe3.slice/crio-84a91ca8d02262bd330de692cd55c4de10a03c8817036f740d465918f486660e WatchSource:0}: Error finding container 84a91ca8d02262bd330de692cd55c4de10a03c8817036f740d465918f486660e: Status 404 returned error can't find the container with id 84a91ca8d02262bd330de692cd55c4de10a03c8817036f740d465918f486660e
Oct 31 07:43:49.435130 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.435095    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.435210 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.435172    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.435259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.435209    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.492070 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.492043    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"]
Oct 31 07:43:49.496077 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:49.496020    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b86fad7_2774_4996_9022_532c72f7e325.slice/crio-0522625a1e8807c8b10d237c5f9f39fea7b4d42e0c40432687ec5223426847c0 WatchSource:0}: Error finding container 0522625a1e8807c8b10d237c5f9f39fea7b4d42e0c40432687ec5223426847c0: Status 404 returned error can't find the container with id 0522625a1e8807c8b10d237c5f9f39fea7b4d42e0c40432687ec5223426847c0
Oct 31 07:43:49.500660 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.500637    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:49.502308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.501859    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.502587 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.502360    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.502587 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.502406    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.504929 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.504896    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"]
Oct 31 07:43:49.506859 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:49.506828    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a57dee9_c39b_4039_b990_889f7ecb0d96.slice/crio-d4f619f2454a4b1bd0b4f6751a77ad2abee1961734ce52483be2922323cdf6c8 WatchSource:0}: Error finding container d4f619f2454a4b1bd0b4f6751a77ad2abee1961734ce52483be2922323cdf6c8: Status 404 returned error can't find the container with id d4f619f2454a4b1bd0b4f6751a77ad2abee1961734ce52483be2922323cdf6c8
Oct 31 07:43:49.514215 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.514182    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.514323 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.514269    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.514402 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.514340    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.856738 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.856266    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"]
Oct 31 07:43:49.859109 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:49.859074    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode227c5a8_109d_4570_a1bb_b15de0ac0cfd.slice/crio-6267e08edb1b7f960cfccd77b01f164cd9caa1fdefd9d6540900412179db4014 WatchSource:0}: Error finding container 6267e08edb1b7f960cfccd77b01f164cd9caa1fdefd9d6540900412179db4014: Status 404 returned error can't find the container with id 6267e08edb1b7f960cfccd77b01f164cd9caa1fdefd9d6540900412179db4014
Oct 31 07:43:49.862918 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.862259    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.862918 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.862451    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.862918 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.862497    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:43:49.928371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.928238    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" event={"ID":"8a57dee9-c39b-4039-b990-889f7ecb0d96","Type":"ContainerStarted","Data":"1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00"}
Oct 31 07:43:49.928371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.928319    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" event={"ID":"8a57dee9-c39b-4039-b990-889f7ecb0d96","Type":"ContainerStarted","Data":"d4f619f2454a4b1bd0b4f6751a77ad2abee1961734ce52483be2922323cdf6c8"}
Oct 31 07:43:49.932573 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.931965    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" event={"ID":"8b86fad7-2774-4996-9022-532c72f7e325","Type":"ContainerStarted","Data":"ccc51f9f825c48df168bd98230a9cd0f059007724033926cc9bb4645e269f55c"}
Oct 31 07:43:49.932573 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.932025    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" event={"ID":"8b86fad7-2774-4996-9022-532c72f7e325","Type":"ContainerStarted","Data":"0522625a1e8807c8b10d237c5f9f39fea7b4d42e0c40432687ec5223426847c0"}
Oct 31 07:43:49.936079 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.935973    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" event={"ID":"07174d73-53ba-408f-bf2f-6f4273be8fe3","Type":"ContainerStarted","Data":"6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325"}
Oct 31 07:43:49.936079 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.936040    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" event={"ID":"07174d73-53ba-408f-bf2f-6f4273be8fe3","Type":"ContainerStarted","Data":"84a91ca8d02262bd330de692cd55c4de10a03c8817036f740d465918f486660e"}
Oct 31 07:43:49.937791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:49.937714    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" event={"ID":"e227c5a8-109d-4570-a1bb-b15de0ac0cfd","Type":"ContainerStarted","Data":"6267e08edb1b7f960cfccd77b01f164cd9caa1fdefd9d6540900412179db4014"}
Oct 31 07:43:50.034350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.034177    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podStartSLOduration=2.034155952 podStartE2EDuration="2.034155952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:50.032554009 +0000 UTC m=+894.171714625" watchObservedRunningTime="2026-10-31 07:43:50.034155952 +0000 UTC m=+894.173316541"
Oct 31 07:43:50.041387 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.041315    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podStartSLOduration=2.0412962119999998 podStartE2EDuration="2.041296212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:49.984833807 +0000 UTC m=+894.123994396" watchObservedRunningTime="2026-10-31 07:43:50.041296212 +0000 UTC m=+894.180456803"
Oct 31 07:43:50.090104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.090044    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podStartSLOduration=2.090023346 podStartE2EDuration="2.090023346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:50.080631107 +0000 UTC m=+894.219791695" watchObservedRunningTime="2026-10-31 07:43:50.090023346 +0000 UTC m=+894.229183934"
Oct 31 07:43:50.095370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.091558    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:50.095554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.095451    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 07:43:50.095554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.095517    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 07:43:50.125066 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.125034    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:50.125887 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.125854    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 07:43:50.125992 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.125919    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 07:43:50.131465 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.131383    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:50.132682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.132656    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 07:43:50.132780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.132708    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 07:43:50.978308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:50.978233    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" event={"ID":"e227c5a8-109d-4570-a1bb-b15de0ac0cfd","Type":"ContainerStarted","Data":"21e40bc21069aaa1fb280c7c64872677d9e46b547c6d94ce6e9ce30bb038f78e"}
Oct 31 07:43:51.103350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.096743    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 07:43:51.103350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.096811    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 07:43:51.170695 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.170655    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 07:43:51.178306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.174347    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 07:43:51.199308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.191557    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 07:43:51.199308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.191619    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 07:43:51.504693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.504650    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 07:43:51.504980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.504945    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 07:43:51.505464 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.505126    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:51.879478 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.879327    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podStartSLOduration=2.879303422 podStartE2EDuration="2.879303422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:51.072616759 +0000 UTC m=+895.211777348" watchObservedRunningTime="2026-10-31 07:43:51.879303422 +0000 UTC m=+896.018464011"
Oct 31 07:43:51.900693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.900654    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"]
Oct 31 07:43:51.926341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.926308    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:51.954444 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.936653    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"]
Oct 31 07:43:51.986700 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.986666    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:43:52.154645 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.989885    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:43:52.154898 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.990003    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:43:52.155010 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.990150    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:43:52.155118 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:51.990198    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9pzm\" (UniqueName: \"kubernetes.io/projected/230e6b54-1f3b-427b-9e5d-bd998203856d-kube-api-access-l9pzm\") pod \"maas-api-revoke-sub-b91fa08f-8h6gg\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.155335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.155314    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91fa08f-8h6gg\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.190335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.102419    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 07:43:52.190335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.174407    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 07:43:52.190335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.131815    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 07:43:52.190335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.174486    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 07:43:52.190335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.132612    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 07:43:52.190335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.174910    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 07:43:52.190335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.174983    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.191275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.191231    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91fa08f-8h6gg\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.390011    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.391024    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.431508    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9pzm\" (UniqueName: \"kubernetes.io/projected/230e6b54-1f3b-427b-9e5d-bd998203856d-kube-api-access-l9pzm\") pod \"maas-api-revoke-sub-b91fa08f-8h6gg\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.431581    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91fa08f-8h6gg\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.431726    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91fa08f-8h6gg\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.452060    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b91fa08f-8h6gg\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.465421    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b91fa08f-8h6gg\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.465528    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.465919    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.466526    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.478368    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.513341    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.513404    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.543606    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.548361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.547007    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9pzm\" (UniqueName: \"kubernetes.io/projected/230e6b54-1f3b-427b-9e5d-bd998203856d-kube-api-access-l9pzm\") pod \"maas-api-revoke-sub-b91fa08f-8h6gg\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.575307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.569393    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.575307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.569512    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.577966 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.577923    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:52.612823 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.608815    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.694814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.694773    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:52.827108 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.827065    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded"
Oct 31 07:43:52.880312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:52.869387    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:53.077723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.077386    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2khk6" podUID="67ce1496-b71b-4aca-a7f5-ac8dbd09b5fe" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.22:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:53.526722 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.507899    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 07:43:53.526722 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.508310    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 07:43:53.653312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.535398    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": context deadline exceeded"
Oct 31 07:43:53.653312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.570448    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:53.653312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.572374    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:53.653312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.579372    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:53.653312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.622303    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 07:43:53.686314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.668510    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 07:43:53.699307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.690551    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 07:43:53.710417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.706407    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 07:43:53.863220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.725299    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 31 07:43:53.925342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.680101    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 07:43:53.925342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.899162    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 07:43:54.060337 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.926736    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 07:43:54.060337 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.926818    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 07:43:54.060337 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.929362    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 07:43:54.099557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.099506    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 07:43:54.169941 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.948223    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:43:54.178395 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:53.933988    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 07:43:54.227307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.202218    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 07:43:54.424367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.420394    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.425050    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.643112    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.627185    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.611817    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.425098    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.643965    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.627228    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.627264    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.652200    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.659754    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.660458    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.660513    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.661045    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:54.706363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.661403    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 07:43:54.820639 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.808912    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:54.957368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.955259    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 07:43:54.957368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:54.955744    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 31 07:43:55.117602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.115728    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 07:43:55.117602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.115809    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 07:43:55.143487 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.124655    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 07:43:55.143487 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.124731    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 07:43:55.177497 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.177436    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 07:43:55.177811 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.177761    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 07:43:55.180616 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.180572    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:55.255924 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.255809    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 31 07:43:55.297307 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:43:55.293595    5829 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.142s"
Oct 31 07:43:55.301492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.301461    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"]
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332568    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332633    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332674    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332712    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332746    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332784    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332821    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332854    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332888    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:43:55.352313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.332923    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:43:55.561368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.527107    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 07:43:55.854799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.854754    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 07:43:55.880395 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.659209    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:55.880986 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.643379    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:55.898639 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.659245    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 31 07:43:55.944306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.659266    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 07:43:56.175348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.667644    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 07:43:56.191728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.667665    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 07:43:56.301348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.772100    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": context deadline exceeded"
Oct 31 07:43:56.301601 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.809844    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:56.488817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:55.667608    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:56.490716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.004934    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 07:43:56.517510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.004971    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 31 07:43:56.518880 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.225391    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:56.520048 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.226577    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 07:43:56.520225 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.520193    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 07:43:56.520369 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.226622    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 07:43:56.520488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.520467    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 07:43:56.520589 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.273971    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 07:43:56.520682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.520666    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 07:43:56.643107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.643069    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 07:43:56.643235 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.643136    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 07:43:56.914370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.880815    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:56.914370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.897442    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:56.977713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:56.977663    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:57.095202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.093683    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body=
Oct 31 07:43:57.095202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.093746    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused"
Oct 31 07:43:57.187411 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.160440    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:57.187411 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.170415    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:57.211600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.211565    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"]
Oct 31 07:43:57.392125 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.391937    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 07:43:57.484292 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.484132    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 07:43:57.484540 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.484507    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 07:43:57.489815 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.489470    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:57.532315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.530864    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 07:43:57.532315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.530981    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 07:43:57.554580 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.554534    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:57.566376 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.562382    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:43:57.566731 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.566709    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.567679 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.566632    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:43:57.567874 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.567845    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"]
Oct 31 07:43:57.569081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.568594    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:43:57.569081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.568657    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:43:57.569081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.568707    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:43:57.569081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.568752    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:43:57.570274 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.570239    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:43:57.570423 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.570388    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:43:57.576429 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.576391    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 07:43:57.576573 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.576459    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 07:43:57.576573 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.576398    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg" event={"ID":"230e6b54-1f3b-427b-9e5d-bd998203856d","Type":"ContainerStarted","Data":"e07b4e84f525923a8b88d1f2a8eb4d0483a72ad766eaca6fc74eeaf4a9e47ff0"}
Oct 31 07:43:57.595986 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.584190    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:43:57.599398 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.599363    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:43:57.686939 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.686910    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:43:57.691093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.691062    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:43:57.691373 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.691340    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82k7t\" (UniqueName: \"kubernetes.io/projected/c39ec502-6388-452a-a67a-b1f7be7d0319-kube-api-access-82k7t\") pod \"maas-api-revoke-sub-b69cdb34-2zbbp\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.695343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.695110    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b69cdb34-2zbbp\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.695472 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.695371    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b69cdb34-2zbbp\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.802383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.802216    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b69cdb34-2zbbp\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.802807 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.802773    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82k7t\" (UniqueName: \"kubernetes.io/projected/c39ec502-6388-452a-a67a-b1f7be7d0319-kube-api-access-82k7t\") pod \"maas-api-revoke-sub-b69cdb34-2zbbp\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.802942 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.802841    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b69cdb34-2zbbp\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.805417 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:43:57.805383    5829 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230e6b54_1f3b_427b_9e5d_bd998203856d.slice/crio-24794dc39f8123f060150c51a529499062fdccd67572d47a42f9687733565972.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 07:43:57.816602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.806609    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b69cdb34-2zbbp\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.816602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.808522    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b69cdb34-2zbbp\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.838356 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.838324    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82k7t\" (UniqueName: \"kubernetes.io/projected/c39ec502-6388-452a-a67a-b1f7be7d0319-kube-api-access-82k7t\") pod \"maas-api-revoke-sub-b69cdb34-2zbbp\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:57.975441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:57.975307    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:43:58.004092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.004066    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:43:58.023312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.016927    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:43:58.126916 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.126719    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 31 07:43:58.126916 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.126787    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 31 07:43:58.134963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.134932    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 31 07:43:58.135325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.135266    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 31 07:43:58.506450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.506412    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 31 07:43:58.507188 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.507156    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 31 07:43:58.604191 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.604133    5829 generic.go:356] "Generic (PLEG): container finished" podID="230e6b54-1f3b-427b-9e5d-bd998203856d" containerID="24794dc39f8123f060150c51a529499062fdccd67572d47a42f9687733565972" exitCode=0
Oct 31 07:43:58.604534 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.604502    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg" event={"ID":"230e6b54-1f3b-427b-9e5d-bd998203856d","Type":"ContainerDied","Data":"24794dc39f8123f060150c51a529499062fdccd67572d47a42f9687733565972"}
Oct 31 07:43:58.699072 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:43:58.698742    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc39ec502_6388_452a_a67a_b1f7be7d0319.slice/crio-74d9df0aee78185c8f2a096f36cc9c6721bb34c267de93f1be4e57f325a16439 WatchSource:0}: Error finding container 74d9df0aee78185c8f2a096f36cc9c6721bb34c267de93f1be4e57f325a16439: Status 404 returned error can't find the container with id 74d9df0aee78185c8f2a096f36cc9c6721bb34c267de93f1be4e57f325a16439
Oct 31 07:43:58.721471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.721422    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"]
Oct 31 07:43:58.951625 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.951589    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:58.953798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.953756    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:58.960966 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:58.960899    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:43:59.140994 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.140963    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:59.142923 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.142895    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:59.143434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.143414    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:43:59.336555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.336265    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:59.337501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.337474    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:59.338541 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.338519    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:43:59.507448 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.507362    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:59.507950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.507928    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:59.508832 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.508812    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:43:59.626447 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.626400    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp" event={"ID":"c39ec502-6388-452a-a67a-b1f7be7d0319","Type":"ContainerStarted","Data":"875a969d99ece404246aa14bae80252eaebc3c8119fc467035f32ac407b4e575"}
Oct 31 07:43:59.626687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.626543    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp" event={"ID":"c39ec502-6388-452a-a67a-b1f7be7d0319","Type":"ContainerStarted","Data":"74d9df0aee78185c8f2a096f36cc9c6721bb34c267de93f1be4e57f325a16439"}
Oct 31 07:43:59.671462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.671408    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp" podStartSLOduration=4.671390881 podStartE2EDuration="4.671390881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:59.664931751 +0000 UTC m=+903.804092341" watchObservedRunningTime="2026-10-31 07:43:59.671390881 +0000 UTC m=+903.810551497"
Oct 31 07:43:59.819988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.819960    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:43:59.860777 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.860742    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-openshift-ca\") pod \"230e6b54-1f3b-427b-9e5d-bd998203856d\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") "
Oct 31 07:43:59.860990 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.860968    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230e6b54-1f3b-427b-9e5d-bd998203856d-kube-api-access-l9pzm\" (UniqueName: \"kubernetes.io/projected/230e6b54-1f3b-427b-9e5d-bd998203856d-kube-api-access-l9pzm\") pod \"230e6b54-1f3b-427b-9e5d-bd998203856d\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") "
Oct 31 07:43:59.861130 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.861103    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-openshift-ca" pod "230e6b54-1f3b-427b-9e5d-bd998203856d" (UID: "230e6b54-1f3b-427b-9e5d-bd998203856d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:43:59.861344 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.861322    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-xks-ca\") pod \"230e6b54-1f3b-427b-9e5d-bd998203856d\" (UID: \"230e6b54-1f3b-427b-9e5d-bd998203856d\") "
Oct 31 07:43:59.862017 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.861993    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:59.863522 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.863499    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230e6b54-1f3b-427b-9e5d-bd998203856d-kube-api-access-l9pzm" pod "230e6b54-1f3b-427b-9e5d-bd998203856d" (UID: "230e6b54-1f3b-427b-9e5d-bd998203856d"). InnerVolumeSpecName "kube-api-access-l9pzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:43:59.864254 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.864230    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-xks-ca" pod "230e6b54-1f3b-427b-9e5d-bd998203856d" (UID: "230e6b54-1f3b-427b-9e5d-bd998203856d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:43:59.964081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.964040    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9pzm\" (UniqueName: \"kubernetes.io/projected/230e6b54-1f3b-427b-9e5d-bd998203856d-kube-api-access-l9pzm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:43:59.964081 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:43:59.964078    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/230e6b54-1f3b-427b-9e5d-bd998203856d-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:00.636381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:00.636345    5829 generic.go:356] "Generic (PLEG): container finished" podID="c39ec502-6388-452a-a67a-b1f7be7d0319" containerID="875a969d99ece404246aa14bae80252eaebc3c8119fc467035f32ac407b4e575" exitCode=0
Oct 31 07:44:00.636816 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:00.636435    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp" event={"ID":"c39ec502-6388-452a-a67a-b1f7be7d0319","Type":"ContainerDied","Data":"875a969d99ece404246aa14bae80252eaebc3c8119fc467035f32ac407b4e575"}
Oct 31 07:44:00.638013 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:00.637981    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg" event={"ID":"230e6b54-1f3b-427b-9e5d-bd998203856d","Type":"ContainerDied","Data":"e07b4e84f525923a8b88d1f2a8eb4d0483a72ad766eaca6fc74eeaf4a9e47ff0"}
Oct 31 07:44:00.638139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:00.638006    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"
Oct 31 07:44:00.638139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:00.638036    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e07b4e84f525923a8b88d1f2a8eb4d0483a72ad766eaca6fc74eeaf4a9e47ff0"
Oct 31 07:44:01.800980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.800952    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:44:01.895924 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.895889    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-openshift-ca\") pod \"c39ec502-6388-452a-a67a-b1f7be7d0319\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") "
Oct 31 07:44:01.896095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.895981    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-xks-ca\") pod \"c39ec502-6388-452a-a67a-b1f7be7d0319\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") "
Oct 31 07:44:01.896095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.896067    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c39ec502-6388-452a-a67a-b1f7be7d0319-kube-api-access-82k7t\" (UniqueName: \"kubernetes.io/projected/c39ec502-6388-452a-a67a-b1f7be7d0319-kube-api-access-82k7t\") pod \"c39ec502-6388-452a-a67a-b1f7be7d0319\" (UID: \"c39ec502-6388-452a-a67a-b1f7be7d0319\") "
Oct 31 07:44:01.896389 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.896363    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-openshift-ca" pod "c39ec502-6388-452a-a67a-b1f7be7d0319" (UID: "c39ec502-6388-452a-a67a-b1f7be7d0319"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:01.896796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.896756    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:01.898202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.898175    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-xks-ca" pod "c39ec502-6388-452a-a67a-b1f7be7d0319" (UID: "c39ec502-6388-452a-a67a-b1f7be7d0319"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:01.898301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.898244    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c39ec502-6388-452a-a67a-b1f7be7d0319-kube-api-access-82k7t" pod "c39ec502-6388-452a-a67a-b1f7be7d0319" (UID: "c39ec502-6388-452a-a67a-b1f7be7d0319"). InnerVolumeSpecName "kube-api-access-82k7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:01.997911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.997879    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c39ec502-6388-452a-a67a-b1f7be7d0319-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:01.997911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:01.997909    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82k7t\" (UniqueName: \"kubernetes.io/projected/c39ec502-6388-452a-a67a-b1f7be7d0319-kube-api-access-82k7t\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:02.656430 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:02.656400    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"
Oct 31 07:44:02.656588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:02.656431    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp" event={"ID":"c39ec502-6388-452a-a67a-b1f7be7d0319","Type":"ContainerDied","Data":"74d9df0aee78185c8f2a096f36cc9c6721bb34c267de93f1be4e57f325a16439"}
Oct 31 07:44:02.656588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:02.656484    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d9df0aee78185c8f2a096f36cc9c6721bb34c267de93f1be4e57f325a16439"
Oct 31 07:44:03.981949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:03.981910    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"]
Oct 31 07:44:03.985577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:03.983464    5829 cpu_manager.go:414] "containerMap: removing container" podUID="c39ec502-6388-452a-a67a-b1f7be7d0319" containerName="revoke-keys"
Oct 31 07:44:03.985577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:03.983490    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c39ec502-6388-452a-a67a-b1f7be7d0319" containerName="revoke-keys"
Oct 31 07:44:03.985577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:03.983599    5829 cpu_manager.go:414] "containerMap: removing container" podUID="230e6b54-1f3b-427b-9e5d-bd998203856d" containerName="revoke-keys"
Oct 31 07:44:03.985577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:03.983608    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230e6b54-1f3b-427b-9e5d-bd998203856d" containerName="revoke-keys"
Oct 31 07:44:03.985577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:03.983902    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="c39ec502-6388-452a-a67a-b1f7be7d0319" containerName="revoke-keys"
Oct 31 07:44:03.985577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:03.983950    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="230e6b54-1f3b-427b-9e5d-bd998203856d" containerName="revoke-keys"
Oct 31 07:44:03.989926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:03.989900    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.002376 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.002349    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"]
Oct 31 07:44:04.087966 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.087932    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"]
Oct 31 07:44:04.093000 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.092972    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.123970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.123936    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"]
Oct 31 07:44:04.130164 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.130118    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-data\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.130662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.130629    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-certs\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.130796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.130678    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x49d6\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-kube-api-access-x49d6\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.130796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.130762    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-token\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.131346 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.130802    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-podinfo\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.131346 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.130837    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-credential-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.131346 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.130907    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istiod-ca-cert\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.132080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.132051    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-envoy\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.132178 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.132111    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.233805 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.233721    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-token\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.233956 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.233844    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-certs\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.233956 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.233881    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x49d6\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-kube-api-access-x49d6\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.233956 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.233935    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4pl\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-kube-api-access-5b4pl\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.234071 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.233965    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-token\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234071 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234046    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-podinfo\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234138 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234091    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-credential-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234181 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234157    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-credential-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.234227 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234195    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istiod-ca-cert\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234274 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234254    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-certs\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234399 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234379    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-envoy\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234449 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234426    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.234514 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234462    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-certs\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.234569 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234507    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-credential-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234627 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234569    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234632    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-podinfo\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.234773 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234748    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c50054-e3cc-4286-b59f-0a012f93fa92-istiod-ca-cert\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.234834 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234818    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-data\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234894 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234870    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-socket\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.234959 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234938    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-envoy\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.234999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.234972    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-data\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.235042 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.235008    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istiod-ca-cert\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.235173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.235153    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-data\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.236595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.236574    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-envoy\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.236678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.236659    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-podinfo\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.254186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.254157    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-token\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.254485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.254463    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x49d6\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-kube-api-access-x49d6\") pod \"e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") " pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.318062 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.318020    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:04.336134 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336101    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-credential-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336213    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336248    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-certs\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336464    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-podinfo\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336490    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-credential-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336572    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c50054-e3cc-4286-b59f-0a012f93fa92-istiod-ca-cert\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336598    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-socket\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336702    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-certs\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336686    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-envoy\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336835 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336761    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-data\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336835 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336827    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-token\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.336979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.336962    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4pl\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-kube-api-access-5b4pl\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.337240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.337191    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-data\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.337353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.337312    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c50054-e3cc-4286-b59f-0a012f93fa92-istiod-ca-cert\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.339422 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.339394    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-envoy\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.339715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.339686    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-podinfo\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.356694 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.356662    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-token\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.357158 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.357136    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4pl\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-kube-api-access-5b4pl\") pod \"e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") " pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.410974 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.410937    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:04.650297 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:04.648917    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e57edca_955c_4caa_ba2f_4c58e3be16a4.slice/crio-0f1d534ba015bc63a101eb7502bee4e4d629624f4b61dd8489c0ebb754877884 WatchSource:0}: Error finding container 0f1d534ba015bc63a101eb7502bee4e4d629624f4b61dd8489c0ebb754877884: Status 404 returned error can't find the container with id 0f1d534ba015bc63a101eb7502bee4e4d629624f4b61dd8489c0ebb754877884
Oct 31 07:44:04.650297 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.649900    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"]
Oct 31 07:44:04.652262 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.652222    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:04.652379 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.652358    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:04.652435 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.652407    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:04.677815 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.677782    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" event={"ID":"7e57edca-955c-4caa-ba2f-4c58e3be16a4","Type":"ContainerStarted","Data":"0f1d534ba015bc63a101eb7502bee4e4d629624f4b61dd8489c0ebb754877884"}
Oct 31 07:44:04.734215 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.734150    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"]
Oct 31 07:44:04.735093 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:04.735067    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c50054_e3cc_4286_b59f_0a012f93fa92.slice/crio-f8281f3a07b5d946903b93ad18d2bd306d16488858f6202994dd8947544b1d2e WatchSource:0}: Error finding container f8281f3a07b5d946903b93ad18d2bd306d16488858f6202994dd8947544b1d2e: Status 404 returned error can't find the container with id f8281f3a07b5d946903b93ad18d2bd306d16488858f6202994dd8947544b1d2e
Oct 31 07:44:04.737943 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.737909    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:04.737998 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.737980    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:04.738030 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:04.738010    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:05.242300 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.242155    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"]
Oct 31 07:44:05.253315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.250153    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.268979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.268943    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:44:05.269144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.269062    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:44:05.269144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.269094    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:44:05.269835 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.269811    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:44:05.270473 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.270448    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"]
Oct 31 07:44:05.354368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.354336    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9bf\" (UniqueName: \"kubernetes.io/projected/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-kube-api-access-7r9bf\") pod \"maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.354534 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.354389    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.354707 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.354685    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.456640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.456605    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.456795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.456705    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9bf\" (UniqueName: \"kubernetes.io/projected/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-kube-api-access-7r9bf\") pod \"maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.456795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.456739    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.457424 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.457399    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.459174 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.459145    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.478327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.478274    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9bf\" (UniqueName: \"kubernetes.io/projected/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-kube-api-access-7r9bf\") pod \"maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.590102 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.590022    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:05.688970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.688935    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" event={"ID":"a8c50054-e3cc-4286-b59f-0a012f93fa92","Type":"ContainerStarted","Data":"2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01"}
Oct 31 07:44:05.689178 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.689164    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" event={"ID":"a8c50054-e3cc-4286-b59f-0a012f93fa92","Type":"ContainerStarted","Data":"f8281f3a07b5d946903b93ad18d2bd306d16488858f6202994dd8947544b1d2e"}
Oct 31 07:44:05.691710 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.691679    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" event={"ID":"7e57edca-955c-4caa-ba2f-4c58e3be16a4","Type":"ContainerStarted","Data":"dc707bdc4b5ae066eb5f7799d82aeecce2f909ccb238d61b37f29c60e5351fb3"}
Oct 31 07:44:05.879548 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.879429    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" podStartSLOduration=1.8794096040000001 podStartE2EDuration="1.879409604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:05.762712721 +0000 UTC m=+909.901873310" watchObservedRunningTime="2026-10-31 07:44:05.879409604 +0000 UTC m=+910.018570193"
Oct 31 07:44:05.883015 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:05.882961    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podStartSLOduration=2.882949333 podStartE2EDuration="2.882949333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:05.810437454 +0000 UTC m=+909.949598043" watchObservedRunningTime="2026-10-31 07:44:05.882949333 +0000 UTC m=+910.022109922"
Oct 31 07:44:06.059679 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:06.059557    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41dcd454_0846_4f21_a3dd_fe4dc26e26ef.slice/crio-7a92524f67c857860d75a133358ba534355faae79e273e36796f815729402f84 WatchSource:0}: Error finding container 7a92524f67c857860d75a133358ba534355faae79e273e36796f815729402f84: Status 404 returned error can't find the container with id 7a92524f67c857860d75a133358ba534355faae79e273e36796f815729402f84
Oct 31 07:44:06.059977 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.059880    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"]
Oct 31 07:44:06.322315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.321646    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:06.322315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.321788    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 07:44:06.322315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.321831    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 07:44:06.425305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.411450    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:06.425305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.419895    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 07:44:06.425305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.419956    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 07:44:06.701868 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.701827    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x" event={"ID":"41dcd454-0846-4f21-a3dd-fe4dc26e26ef","Type":"ContainerStarted","Data":"3451b4a9f7f1e89209d511822f52de7ed5a2d26bf9ddc9a6529c1f98e10c5c6b"}
Oct 31 07:44:06.702108 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.701893    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x" event={"ID":"41dcd454-0846-4f21-a3dd-fe4dc26e26ef","Type":"ContainerStarted","Data":"7a92524f67c857860d75a133358ba534355faae79e273e36796f815729402f84"}
Oct 31 07:44:06.759845 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:06.759769    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x" podStartSLOduration=1.759745561 podStartE2EDuration="1.759745561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:06.756700663 +0000 UTC m=+910.895861246" watchObservedRunningTime="2026-10-31 07:44:06.759745561 +0000 UTC m=+910.898906150"
Oct 31 07:44:07.330315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:07.325785    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 07:44:07.330315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:07.325849    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 07:44:07.426415 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:07.412808    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 07:44:07.426415 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:07.412887    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 07:44:07.754463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:07.753362    5829 generic.go:356] "Generic (PLEG): container finished" podID="41dcd454-0846-4f21-a3dd-fe4dc26e26ef" containerID="3451b4a9f7f1e89209d511822f52de7ed5a2d26bf9ddc9a6529c1f98e10c5c6b" exitCode=0
Oct 31 07:44:07.754805 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:07.754765    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x" event={"ID":"41dcd454-0846-4f21-a3dd-fe4dc26e26ef","Type":"ContainerDied","Data":"3451b4a9f7f1e89209d511822f52de7ed5a2d26bf9ddc9a6529c1f98e10c5c6b"}
Oct 31 07:44:08.335304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:08.322562    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 07:44:08.335304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:08.322625    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 07:44:08.422353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:08.415682    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 07:44:08.422353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:08.415747    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 07:44:09.272951 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.272919    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:09.364439 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.363432    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 07:44:09.364439 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.363513    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 07:44:09.369608 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.369574    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-openshift-ca\") pod \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") "
Oct 31 07:44:09.369781 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.369769    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-kube-api-access-7r9bf\" (UniqueName: \"kubernetes.io/projected/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-kube-api-access-7r9bf\") pod \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") "
Oct 31 07:44:09.369865 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.369826    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-xks-ca\") pod \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\" (UID: \"41dcd454-0846-4f21-a3dd-fe4dc26e26ef\") "
Oct 31 07:44:09.371162 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.370935    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-openshift-ca" pod "41dcd454-0846-4f21-a3dd-fe4dc26e26ef" (UID: "41dcd454-0846-4f21-a3dd-fe4dc26e26ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:09.375220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.375186    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-kube-api-access-7r9bf" pod "41dcd454-0846-4f21-a3dd-fe4dc26e26ef" (UID: "41dcd454-0846-4f21-a3dd-fe4dc26e26ef"). InnerVolumeSpecName "kube-api-access-7r9bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:09.376849 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.376810    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-xks-ca" pod "41dcd454-0846-4f21-a3dd-fe4dc26e26ef" (UID: "41dcd454-0846-4f21-a3dd-fe4dc26e26ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:09.445365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.411735    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 07:44:09.445679 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.445631    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 07:44:09.472159 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.472125    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9bf\" (UniqueName: \"kubernetes.io/projected/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-kube-api-access-7r9bf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:09.472159 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.472155    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:09.472381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.472170    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dcd454-0846-4f21-a3dd-fe4dc26e26ef-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:09.871164 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.871082    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x" event={"ID":"41dcd454-0846-4f21-a3dd-fe4dc26e26ef","Type":"ContainerDied","Data":"7a92524f67c857860d75a133358ba534355faae79e273e36796f815729402f84"}
Oct 31 07:44:09.871164 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.871133    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a92524f67c857860d75a133358ba534355faae79e273e36796f815729402f84"
Oct 31 07:44:09.871357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.871189    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"
Oct 31 07:44:09.964603 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:09.964527    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"]
Oct 31 07:44:10.324316 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:10.323756    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 07:44:10.324316 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:10.323837    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 07:44:10.413305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:10.411648    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 07:44:10.413305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:10.411712    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 07:44:10.762723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:10.762691    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:44:11.318491 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:11.318446    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body=
Oct 31 07:44:11.318721 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:11.318526    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused"
Oct 31 07:44:11.411492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:11.411441    5829 patch_prober.go:29] interesting pod/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 31 07:44:11.411674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:11.411523    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 31 07:44:12.984766 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:12.984720    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:12.987430 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:12.986210    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:12.987550 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:12.987521    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:13.363969 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:13.363873    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:44:13.364239 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:13.364206    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:14.435628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:14.435590    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:44:14.436325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:14.436266    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:14.802621 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:14.802517    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:14.908349 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:14.897350    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:14.923532 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:14.923462    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 31 07:44:14.925315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:14.925245    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.212355 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.212205    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 07:44:15.247347 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.247302    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 31 07:44:15.384959 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.384918    5829 patch_prober.go:29] interesting pod/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:44:15.385308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.385242    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.486498 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.486389    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.497492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.497451    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:44:15.497804 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.497755    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.519485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.519433    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.534241 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.533995    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.534738 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.534523    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.587318 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.587181    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.671778 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.671736    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.690144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.690087    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.807804 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.807671    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.808591 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.808567    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:44:15.907314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.901056    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:15.956426 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.956372    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 31 07:44:16.084373 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:15.965160    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:16.178736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.178700    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:44:16.271367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.271337    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:44:16.297309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.261171    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 07:44:16.521457 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.506108    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:16.656260 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.525394    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Oct 31 07:44:16.756119 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.542203    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Oct 31 07:44:16.772509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.562769    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:16.860632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.860579    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:16.902305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.872634    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 07:44:16.902305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:16.594543    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:17.151304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.148687    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main" probeResult="failure" output="Get \"https://10.134.0.87:8000/ready\": context deadline exceeded"
Oct 31 07:44:17.217988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.207299    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:17.267107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.267047    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:17.502388    5829 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.227s"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.502589    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.502784    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.502851    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.502958    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.502992    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"]
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.503085    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.503148    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.503206    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.503266    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.503344    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:44:17.517604 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.514958    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:17.531357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.522441    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:44:17.531357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.528164    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:17.531357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.530499    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 07:44:17.612244 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:17.612205    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:44:17.745994 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.745947    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:17.783439 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.783104    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:17.783728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.783695    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:17.862697 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.862402    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:17.920581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.920536    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:17.927906 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.927367    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:44:17.944663 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.932542    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 07:44:17.946968 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:17.946939    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:44:18.069544 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.069459    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:44:18.069815 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.069797    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:44:18.107265 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.106406    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:44:18.183537 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.182815    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:44:18.207115 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.202086    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:44:18.241416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.241362    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:18.254345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.251060    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:44:18.279618 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.279592    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:44:18.290221 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.290175    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:18.291839 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.291742    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:44:18.302812 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:18.302790    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:44:22.457742 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.457706    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"]
Oct 31 07:44:22.459832 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.459792    5829 cpu_manager.go:414] "containerMap: removing container" podUID="41dcd454-0846-4f21-a3dd-fe4dc26e26ef" containerName="revoke-keys"
Oct 31 07:44:22.459832 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.459831    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41dcd454-0846-4f21-a3dd-fe4dc26e26ef" containerName="revoke-keys"
Oct 31 07:44:22.460342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.460322    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="41dcd454-0846-4f21-a3dd-fe4dc26e26ef" containerName="revoke-keys"
Oct 31 07:44:22.466659 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.466632    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.483445 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.483420    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"]
Oct 31 07:44:22.490326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.488595    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:44:22.490326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.488651    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:44:22.490326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.488710    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:44:22.495722 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.495702    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:44:22.511694 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.511630    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v" podUID="c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" containerName="payload-pre-processing" containerID="cri-o://f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80" gracePeriod=25
Oct 31 07:44:22.536876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.536840    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4cn\" (UniqueName: \"kubernetes.io/projected/0325aeb1-f4be-4a94-a3c2-c08937d33c00-kube-api-access-lh4cn\") pod \"maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.536876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.536886    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.537092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.537074    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.638978 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.638936    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.639155 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.639127    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4cn\" (UniqueName: \"kubernetes.io/projected/0325aeb1-f4be-4a94-a3c2-c08937d33c00-kube-api-access-lh4cn\") pod \"maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.639242 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.639167    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.639675 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.639652    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.641568 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.641543    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.659389 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.659362    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4cn\" (UniqueName: \"kubernetes.io/projected/0325aeb1-f4be-4a94-a3c2-c08937d33c00-kube-api-access-lh4cn\") pod \"maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.769437 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.769350    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"]
Oct 31 07:44:22.774367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.774347    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.787343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.787314    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:22.792595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.792251    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"]
Oct 31 07:44:22.797677 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.797655    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:22.800223 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.800193    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"]
Oct 31 07:44:22.818257 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.818217    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"]
Oct 31 07:44:22.846328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.841989    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117782a3-bv6xl\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.846328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.842107    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc19f141-6kzlc\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:22.846328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.842133    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc19f141-6kzlc\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:22.846328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.842182    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8p9n\" (UniqueName: \"kubernetes.io/projected/9d208261-0f6b-4417-8b6f-3058381ffb7f-kube-api-access-n8p9n\") pod \"maas-api-revoke-sub-117782a3-bv6xl\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.846328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.842247    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117782a3-bv6xl\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.846328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.842362    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rrrq\" (UniqueName: \"kubernetes.io/projected/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-kube-api-access-2rrrq\") pod \"maas-api-revoke-sub-bc19f141-6kzlc\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:22.943624 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.943588    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc19f141-6kzlc\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:22.943830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.943660    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc19f141-6kzlc\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:22.943830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.943745    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8p9n\" (UniqueName: \"kubernetes.io/projected/9d208261-0f6b-4417-8b6f-3058381ffb7f-kube-api-access-n8p9n\") pod \"maas-api-revoke-sub-117782a3-bv6xl\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.943964 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.943858    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117782a3-bv6xl\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.944025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.944014    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rrrq\" (UniqueName: \"kubernetes.io/projected/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-kube-api-access-2rrrq\") pod \"maas-api-revoke-sub-bc19f141-6kzlc\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:22.944101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.944088    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117782a3-bv6xl\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.944636 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.944606    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc19f141-6kzlc\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:22.944866 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.944841    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117782a3-bv6xl\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.946892 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.946863    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc19f141-6kzlc\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:22.947037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.947013    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117782a3-bv6xl\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.963445 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.963409    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8p9n\" (UniqueName: \"kubernetes.io/projected/9d208261-0f6b-4417-8b6f-3058381ffb7f-kube-api-access-n8p9n\") pod \"maas-api-revoke-sub-117782a3-bv6xl\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:22.963577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:22.963473    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rrrq\" (UniqueName: \"kubernetes.io/projected/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-kube-api-access-2rrrq\") pod \"maas-api-revoke-sub-bc19f141-6kzlc\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:23.092221 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.092138    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:23.116157 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:23.116129    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0325aeb1_f4be_4a94_a3c2_c08937d33c00.slice/crio-855ef1c13eecc18dfe44c96732d18641efa0f95c8a1be1492a2a44837587e958 WatchSource:0}: Error finding container 855ef1c13eecc18dfe44c96732d18641efa0f95c8a1be1492a2a44837587e958: Status 404 returned error can't find the container with id 855ef1c13eecc18dfe44c96732d18641efa0f95c8a1be1492a2a44837587e958
Oct 31 07:44:23.116764 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.116731    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"]
Oct 31 07:44:23.156586 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.156554    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:23.405344 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.405312    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"]
Oct 31 07:44:23.405931 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:23.405901    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d208261_0f6b_4417_8b6f_3058381ffb7f.slice/crio-52b11dabe07261cf416ca463ba402524cc0d6609e233f9f80c4b99d41b967d74 WatchSource:0}: Error finding container 52b11dabe07261cf416ca463ba402524cc0d6609e233f9f80c4b99d41b967d74: Status 404 returned error can't find the container with id 52b11dabe07261cf416ca463ba402524cc0d6609e233f9f80c4b99d41b967d74
Oct 31 07:44:23.494172 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:23.494142    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3878be_78b1_4621_ba23_1b5b4cfb7a72.slice/crio-a67337feb79f707825fcba87cbae3e8e856d2c803d887e55f52b78956bb52f3c WatchSource:0}: Error finding container a67337feb79f707825fcba87cbae3e8e856d2c803d887e55f52b78956bb52f3c: Status 404 returned error can't find the container with id a67337feb79f707825fcba87cbae3e8e856d2c803d887e55f52b78956bb52f3c
Oct 31 07:44:23.495182 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.495154    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"]
Oct 31 07:44:23.831663 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.831576    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp" event={"ID":"0325aeb1-f4be-4a94-a3c2-c08937d33c00","Type":"ContainerStarted","Data":"eac814eefe9771feee095fd46150c57e9ad4cf782745ae1ff6a734c21d49429f"}
Oct 31 07:44:23.831799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.831694    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp" event={"ID":"0325aeb1-f4be-4a94-a3c2-c08937d33c00","Type":"ContainerStarted","Data":"855ef1c13eecc18dfe44c96732d18641efa0f95c8a1be1492a2a44837587e958"}
Oct 31 07:44:23.834025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.833988    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl" event={"ID":"9d208261-0f6b-4417-8b6f-3058381ffb7f","Type":"ContainerStarted","Data":"24955d0eaa9dc57a5670058e6c43f226fd4bd50ce9713d78d54364307bbf82fd"}
Oct 31 07:44:23.834159 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.834055    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl" event={"ID":"9d208261-0f6b-4417-8b6f-3058381ffb7f","Type":"ContainerStarted","Data":"52b11dabe07261cf416ca463ba402524cc0d6609e233f9f80c4b99d41b967d74"}
Oct 31 07:44:23.835522 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.835497    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc" event={"ID":"9e3878be-78b1-4621-ba23-1b5b4cfb7a72","Type":"ContainerStarted","Data":"a67337feb79f707825fcba87cbae3e8e856d2c803d887e55f52b78956bb52f3c"}
Oct 31 07:44:23.872696 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.872636    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp" podStartSLOduration=1.872618798 podStartE2EDuration="1.872618798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:23.869038862 +0000 UTC m=+928.008199447" watchObservedRunningTime="2026-10-31 07:44:23.872618798 +0000 UTC m=+928.011779385"
Oct 31 07:44:23.910903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:23.910810    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl" podStartSLOduration=1.910788358 podStartE2EDuration="1.910788358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:23.906558715 +0000 UTC m=+928.045719305" watchObservedRunningTime="2026-10-31 07:44:23.910788358 +0000 UTC m=+928.049948949"
Oct 31 07:44:24.846383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:24.846271    5829 generic.go:356] "Generic (PLEG): container finished" podID="9d208261-0f6b-4417-8b6f-3058381ffb7f" containerID="24955d0eaa9dc57a5670058e6c43f226fd4bd50ce9713d78d54364307bbf82fd" exitCode=0
Oct 31 07:44:24.846383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:24.846336    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl" event={"ID":"9d208261-0f6b-4417-8b6f-3058381ffb7f","Type":"ContainerDied","Data":"24955d0eaa9dc57a5670058e6c43f226fd4bd50ce9713d78d54364307bbf82fd"}
Oct 31 07:44:24.848317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:24.848265    5829 generic.go:356] "Generic (PLEG): container finished" podID="9e3878be-78b1-4621-ba23-1b5b4cfb7a72" containerID="07bac55930f323462971c97140aedfc0013c55f8b7a04463064bcbc5827f37e4" exitCode=0
Oct 31 07:44:24.848482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:24.848313    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc" event={"ID":"9e3878be-78b1-4621-ba23-1b5b4cfb7a72","Type":"ContainerDied","Data":"07bac55930f323462971c97140aedfc0013c55f8b7a04463064bcbc5827f37e4"}
Oct 31 07:44:24.850252 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:24.850217    5829 generic.go:356] "Generic (PLEG): container finished" podID="0325aeb1-f4be-4a94-a3c2-c08937d33c00" containerID="eac814eefe9771feee095fd46150c57e9ad4cf782745ae1ff6a734c21d49429f" exitCode=0
Oct 31 07:44:24.850390 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:24.850305    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp" event={"ID":"0325aeb1-f4be-4a94-a3c2-c08937d33c00","Type":"ContainerDied","Data":"eac814eefe9771feee095fd46150c57e9ad4cf782745ae1ff6a734c21d49429f"}
Oct 31 07:44:24.988494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:24.988447    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main" containerID="cri-o://2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650" gracePeriod=45
Oct 31 07:44:25.280438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.280407    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:44:25.381528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.381431    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-model-cache\") pod \"d4c3221f-f15b-438b-855d-379088d38324\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") "
Oct 31 07:44:25.381703 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.381567    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-tmp-dir\") pod \"d4c3221f-f15b-438b-855d-379088d38324\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") "
Oct 31 07:44:25.381768 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.381695    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-model-cache" pod "d4c3221f-f15b-438b-855d-379088d38324" (UID: "d4c3221f-f15b-438b-855d-379088d38324"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:25.381768 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.381716    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-dshm\") pod \"d4c3221f-f15b-438b-855d-379088d38324\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") "
Oct 31 07:44:25.381858 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.381805    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c3221f-f15b-438b-855d-379088d38324-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c3221f-f15b-438b-855d-379088d38324-tls-certs\") pod \"d4c3221f-f15b-438b-855d-379088d38324\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") "
Oct 31 07:44:25.381858 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.381826    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-tmp-dir" pod "d4c3221f-f15b-438b-855d-379088d38324" (UID: "d4c3221f-f15b-438b-855d-379088d38324"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:25.381977 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.381961    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c3221f-f15b-438b-855d-379088d38324-kube-api-access-vj66f\" (UniqueName: \"kubernetes.io/projected/d4c3221f-f15b-438b-855d-379088d38324-kube-api-access-vj66f\") pod \"d4c3221f-f15b-438b-855d-379088d38324\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") "
Oct 31 07:44:25.382043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.382029    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-home\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-home\") pod \"d4c3221f-f15b-438b-855d-379088d38324\" (UID: \"d4c3221f-f15b-438b-855d-379088d38324\") "
Oct 31 07:44:25.382455 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.382407    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-home" pod "d4c3221f-f15b-438b-855d-379088d38324" (UID: "d4c3221f-f15b-438b-855d-379088d38324"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:25.382819 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.382798    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:25.382923 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.382825    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:25.382923 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.382838    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:25.385307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.385193    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-dshm" pod "d4c3221f-f15b-438b-855d-379088d38324" (UID: "d4c3221f-f15b-438b-855d-379088d38324"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:25.385307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.385271    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c3221f-f15b-438b-855d-379088d38324-tls-certs" pod "d4c3221f-f15b-438b-855d-379088d38324" (UID: "d4c3221f-f15b-438b-855d-379088d38324"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:25.385451 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.385375    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c3221f-f15b-438b-855d-379088d38324-kube-api-access-vj66f" pod "d4c3221f-f15b-438b-855d-379088d38324" (UID: "d4c3221f-f15b-438b-855d-379088d38324"). InnerVolumeSpecName "kube-api-access-vj66f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:25.484815 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.484775    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj66f\" (UniqueName: \"kubernetes.io/projected/d4c3221f-f15b-438b-855d-379088d38324-kube-api-access-vj66f\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:25.484815 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.484813    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4c3221f-f15b-438b-855d-379088d38324-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:25.485024 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.484826    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4c3221f-f15b-438b-855d-379088d38324-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:25.861124 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.861088    5829 generic.go:356] "Generic (PLEG): container finished" podID="d4c3221f-f15b-438b-855d-379088d38324" containerID="2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650" exitCode=0
Oct 31 07:44:25.861617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.861135    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"
Oct 31 07:44:25.861617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.861182    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" event={"ID":"d4c3221f-f15b-438b-855d-379088d38324","Type":"ContainerDied","Data":"2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650"}
Oct 31 07:44:25.861617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.861238    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp" event={"ID":"d4c3221f-f15b-438b-855d-379088d38324","Type":"ContainerDied","Data":"3de9ad8af10aa68cd10e07c6dab5353b2558a2d7fe4e290636f6e90df929f9b1"}
Oct 31 07:44:25.861617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.861319    5829 scope.go:122] "RemoveContainer" containerID="2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650"
Oct 31 07:44:25.876309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.876269    5829 scope.go:122] "RemoveContainer" containerID="2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650"
Oct 31 07:44:25.876707 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:25.876674    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650\": container with ID starting with 2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650 not found: ID does not exist" containerID="2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650"
Oct 31 07:44:25.876861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.876715    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650"} err="failed to get container status \"2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650\": rpc error: code = NotFound desc = could not find container \"2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650\": container with ID starting with 2c0a873a7f13554dc44600fa326eb947de4fcefe1364654be03f78fecb1f7650 not found: ID does not exist"
Oct 31 07:44:25.918389 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.918341    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"]
Oct 31 07:44:25.935625 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.935536    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-648fbe/distinct-2-w0-648fbe-kserve-75dfb9d6f5-hjpgp"]
Oct 31 07:44:25.982070 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:25.982036    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c3221f-f15b-438b-855d-379088d38324" path="/var/lib/kubelet/pods/d4c3221f-f15b-438b-855d-379088d38324/volumes"
Oct 31 07:44:26.148532 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.148501    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:26.150754 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.150728    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:26.153795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.153769    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:26.300739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.300703    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-xks-ca\") pod \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") "
Oct 31 07:44:26.300943 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.300908    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0325aeb1-f4be-4a94-a3c2-c08937d33c00-kube-api-access-lh4cn\" (UniqueName: \"kubernetes.io/projected/0325aeb1-f4be-4a94-a3c2-c08937d33c00-kube-api-access-lh4cn\") pod \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") "
Oct 31 07:44:26.301009 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.300982    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-xks-ca\") pod \"9d208261-0f6b-4417-8b6f-3058381ffb7f\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") "
Oct 31 07:44:26.301072 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.301032    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-openshift-ca\") pod \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") "
Oct 31 07:44:26.301122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.301081    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d208261-0f6b-4417-8b6f-3058381ffb7f-kube-api-access-n8p9n\" (UniqueName: \"kubernetes.io/projected/9d208261-0f6b-4417-8b6f-3058381ffb7f-kube-api-access-n8p9n\") pod \"9d208261-0f6b-4417-8b6f-3058381ffb7f\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") "
Oct 31 07:44:26.301122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.301110    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-xks-ca\") pod \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\" (UID: \"0325aeb1-f4be-4a94-a3c2-c08937d33c00\") "
Oct 31 07:44:26.301215 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.301131    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-openshift-ca\") pod \"9d208261-0f6b-4417-8b6f-3058381ffb7f\" (UID: \"9d208261-0f6b-4417-8b6f-3058381ffb7f\") "
Oct 31 07:44:26.301418 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.301155    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-openshift-ca\") pod \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") "
Oct 31 07:44:26.301536 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.301508    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-kube-api-access-2rrrq\" (UniqueName: \"kubernetes.io/projected/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-kube-api-access-2rrrq\") pod \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\" (UID: \"9e3878be-78b1-4621-ba23-1b5b4cfb7a72\") "
Oct 31 07:44:26.301905 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.301881    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-openshift-ca" pod "9d208261-0f6b-4417-8b6f-3058381ffb7f" (UID: "9d208261-0f6b-4417-8b6f-3058381ffb7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:26.302468 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.302438    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:26.303306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.303253    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-openshift-ca" pod "0325aeb1-f4be-4a94-a3c2-c08937d33c00" (UID: "0325aeb1-f4be-4a94-a3c2-c08937d33c00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:26.303712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.303684    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-openshift-ca" pod "9e3878be-78b1-4621-ba23-1b5b4cfb7a72" (UID: "9e3878be-78b1-4621-ba23-1b5b4cfb7a72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:26.306921 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.306892    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d208261-0f6b-4417-8b6f-3058381ffb7f-kube-api-access-n8p9n" pod "9d208261-0f6b-4417-8b6f-3058381ffb7f" (UID: "9d208261-0f6b-4417-8b6f-3058381ffb7f"). InnerVolumeSpecName "kube-api-access-n8p9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:26.312316 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.312261    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-xks-ca" pod "0325aeb1-f4be-4a94-a3c2-c08937d33c00" (UID: "0325aeb1-f4be-4a94-a3c2-c08937d33c00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:26.312481 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.312400    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-xks-ca" pod "9e3878be-78b1-4621-ba23-1b5b4cfb7a72" (UID: "9e3878be-78b1-4621-ba23-1b5b4cfb7a72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:26.312540 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.312506    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0325aeb1-f4be-4a94-a3c2-c08937d33c00-kube-api-access-lh4cn" pod "0325aeb1-f4be-4a94-a3c2-c08937d33c00" (UID: "0325aeb1-f4be-4a94-a3c2-c08937d33c00"). InnerVolumeSpecName "kube-api-access-lh4cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:26.313004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.312980    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-kube-api-access-2rrrq" pod "9e3878be-78b1-4621-ba23-1b5b4cfb7a72" (UID: "9e3878be-78b1-4621-ba23-1b5b4cfb7a72"). InnerVolumeSpecName "kube-api-access-2rrrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:26.318965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.318934    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-xks-ca" pod "9d208261-0f6b-4417-8b6f-3058381ffb7f" (UID: "9d208261-0f6b-4417-8b6f-3058381ffb7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:26.404828 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.404793    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh4cn\" (UniqueName: \"kubernetes.io/projected/0325aeb1-f4be-4a94-a3c2-c08937d33c00-kube-api-access-lh4cn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:26.404828 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.404828    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d208261-0f6b-4417-8b6f-3058381ffb7f-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:26.405021 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.404841    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:26.405021 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.404854    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8p9n\" (UniqueName: \"kubernetes.io/projected/9d208261-0f6b-4417-8b6f-3058381ffb7f-kube-api-access-n8p9n\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:26.405021 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.404868    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0325aeb1-f4be-4a94-a3c2-c08937d33c00-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:26.405021 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.404879    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:26.405021 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.404890    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rrrq\" (UniqueName: \"kubernetes.io/projected/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-kube-api-access-2rrrq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:26.405021 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.404902    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3878be-78b1-4621-ba23-1b5b4cfb7a72-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:26.871370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.871242    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"
Oct 31 07:44:26.871370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.871241    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl" event={"ID":"9d208261-0f6b-4417-8b6f-3058381ffb7f","Type":"ContainerDied","Data":"52b11dabe07261cf416ca463ba402524cc0d6609e233f9f80c4b99d41b967d74"}
Oct 31 07:44:26.871849 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.871415    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52b11dabe07261cf416ca463ba402524cc0d6609e233f9f80c4b99d41b967d74"
Oct 31 07:44:26.872885 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.872860    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"
Oct 31 07:44:26.873012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.872861    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc" event={"ID":"9e3878be-78b1-4621-ba23-1b5b4cfb7a72","Type":"ContainerDied","Data":"a67337feb79f707825fcba87cbae3e8e856d2c803d887e55f52b78956bb52f3c"}
Oct 31 07:44:26.873012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.873002    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a67337feb79f707825fcba87cbae3e8e856d2c803d887e55f52b78956bb52f3c"
Oct 31 07:44:26.874394 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.874376    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"
Oct 31 07:44:26.874522 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.874417    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp" event={"ID":"0325aeb1-f4be-4a94-a3c2-c08937d33c00","Type":"ContainerDied","Data":"855ef1c13eecc18dfe44c96732d18641efa0f95c8a1be1492a2a44837587e958"}
Oct 31 07:44:26.874522 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:26.874458    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="855ef1c13eecc18dfe44c96732d18641efa0f95c8a1be1492a2a44837587e958"
Oct 31 07:44:27.156868 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.156832    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"]
Oct 31 07:44:27.749228 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.749193    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"]
Oct 31 07:44:27.749936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.749919    5829 cpu_manager.go:414] "containerMap: removing container" podUID="0325aeb1-f4be-4a94-a3c2-c08937d33c00" containerName="revoke-keys"
Oct 31 07:44:27.749999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.749938    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0325aeb1-f4be-4a94-a3c2-c08937d33c00" containerName="revoke-keys"
Oct 31 07:44:27.749999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.749959    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main"
Oct 31 07:44:27.749999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.749964    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main"
Oct 31 07:44:27.749999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.749972    5829 cpu_manager.go:414] "containerMap: removing container" podUID="9d208261-0f6b-4417-8b6f-3058381ffb7f" containerName="revoke-keys"
Oct 31 07:44:27.749999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.749977    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d208261-0f6b-4417-8b6f-3058381ffb7f" containerName="revoke-keys"
Oct 31 07:44:27.749999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.749991    5829 cpu_manager.go:414] "containerMap: removing container" podUID="9e3878be-78b1-4621-ba23-1b5b4cfb7a72" containerName="revoke-keys"
Oct 31 07:44:27.749999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.749997    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3878be-78b1-4621-ba23-1b5b4cfb7a72" containerName="revoke-keys"
Oct 31 07:44:27.750204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.750121    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3878be-78b1-4621-ba23-1b5b4cfb7a72" containerName="revoke-keys"
Oct 31 07:44:27.750204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.750139    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="0325aeb1-f4be-4a94-a3c2-c08937d33c00" containerName="revoke-keys"
Oct 31 07:44:27.750204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.750148    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c3221f-f15b-438b-855d-379088d38324" containerName="main"
Oct 31 07:44:27.750204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.750156    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d208261-0f6b-4417-8b6f-3058381ffb7f" containerName="revoke-keys"
Oct 31 07:44:27.755603 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.755579    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:27.772777 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.772747    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:44:27.772945 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.772835    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"]
Oct 31 07:44:27.773016 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.772948    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:44:27.773016 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.772956    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:44:27.779487 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.779452    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:44:27.928383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.928344    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c158eef-9j9pc\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:27.928780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.928531    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c158eef-9j9pc\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:27.928877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:27.928854    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcgkx\" (UniqueName: \"kubernetes.io/projected/db35bfba-1b72-4636-b79d-8985b3070ffe-kube-api-access-kcgkx\") pod \"maas-api-revoke-sub-8c158eef-9j9pc\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:28.030779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.030677    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcgkx\" (UniqueName: \"kubernetes.io/projected/db35bfba-1b72-4636-b79d-8985b3070ffe-kube-api-access-kcgkx\") pod \"maas-api-revoke-sub-8c158eef-9j9pc\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:28.030779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.030736    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c158eef-9j9pc\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:28.031032 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.030821    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c158eef-9j9pc\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:28.031481 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.031457    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c158eef-9j9pc\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:28.033424 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.033404    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c158eef-9j9pc\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:28.050718 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.050687    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcgkx\" (UniqueName: \"kubernetes.io/projected/db35bfba-1b72-4636-b79d-8985b3070ffe-kube-api-access-kcgkx\") pod \"maas-api-revoke-sub-8c158eef-9j9pc\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:28.072792 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.072761    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:28.396014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.395989    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"]
Oct 31 07:44:28.895667 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.895630    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc" event={"ID":"db35bfba-1b72-4636-b79d-8985b3070ffe","Type":"ContainerStarted","Data":"57a7face39c027f3a5654ffbad3a86581444deb0169ef2b7022259155372d54c"}
Oct 31 07:44:28.895844 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.895698    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc" event={"ID":"db35bfba-1b72-4636-b79d-8985b3070ffe","Type":"ContainerStarted","Data":"c85b7170e1193d8647d2b1949f61d9b0d1c0d4099d6fe773330fb4e468fca96e"}
Oct 31 07:44:28.937932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:28.937874    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc" podStartSLOduration=1.937859777 podStartE2EDuration="1.937859777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:28.934960051 +0000 UTC m=+933.074120638" watchObservedRunningTime="2026-10-31 07:44:28.937859777 +0000 UTC m=+933.077020365"
Oct 31 07:44:29.906812 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:29.906767    5829 generic.go:356] "Generic (PLEG): container finished" podID="db35bfba-1b72-4636-b79d-8985b3070ffe" containerID="57a7face39c027f3a5654ffbad3a86581444deb0169ef2b7022259155372d54c" exitCode=0
Oct 31 07:44:29.907003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:29.906900    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc" event={"ID":"db35bfba-1b72-4636-b79d-8985b3070ffe","Type":"ContainerDied","Data":"57a7face39c027f3a5654ffbad3a86581444deb0169ef2b7022259155372d54c"}
Oct 31 07:44:31.111192 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.111161    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:31.183868 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.183760    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-xks-ca\") pod \"db35bfba-1b72-4636-b79d-8985b3070ffe\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") "
Oct 31 07:44:31.184048 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.183885    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-openshift-ca\") pod \"db35bfba-1b72-4636-b79d-8985b3070ffe\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") "
Oct 31 07:44:31.184048 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.183980    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db35bfba-1b72-4636-b79d-8985b3070ffe-kube-api-access-kcgkx\" (UniqueName: \"kubernetes.io/projected/db35bfba-1b72-4636-b79d-8985b3070ffe-kube-api-access-kcgkx\") pod \"db35bfba-1b72-4636-b79d-8985b3070ffe\" (UID: \"db35bfba-1b72-4636-b79d-8985b3070ffe\") "
Oct 31 07:44:31.184366 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.184331    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-openshift-ca" pod "db35bfba-1b72-4636-b79d-8985b3070ffe" (UID: "db35bfba-1b72-4636-b79d-8985b3070ffe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:31.184820 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.184792    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:31.186251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.186208    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-xks-ca" pod "db35bfba-1b72-4636-b79d-8985b3070ffe" (UID: "db35bfba-1b72-4636-b79d-8985b3070ffe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:31.186432 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.186409    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db35bfba-1b72-4636-b79d-8985b3070ffe-kube-api-access-kcgkx" pod "db35bfba-1b72-4636-b79d-8985b3070ffe" (UID: "db35bfba-1b72-4636-b79d-8985b3070ffe"). InnerVolumeSpecName "kube-api-access-kcgkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:31.267892 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.267850    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"]
Oct 31 07:44:31.269813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.269782    5829 cpu_manager.go:414] "containerMap: removing container" podUID="db35bfba-1b72-4636-b79d-8985b3070ffe" containerName="revoke-keys"
Oct 31 07:44:31.269992 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.269978    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db35bfba-1b72-4636-b79d-8985b3070ffe" containerName="revoke-keys"
Oct 31 07:44:31.270489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.270466    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="db35bfba-1b72-4636-b79d-8985b3070ffe" containerName="revoke-keys"
Oct 31 07:44:31.278462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.278428    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.285852 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.285820    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac1558a-f5r5q\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.286169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.286147    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxvhg\" (UniqueName: \"kubernetes.io/projected/63062387-4639-49f0-a3f7-6b6150b199ce-kube-api-access-dxvhg\") pod \"maas-api-revoke-sub-cac1558a-f5r5q\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.286368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.286348    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac1558a-f5r5q\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.286573 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.286556    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcgkx\" (UniqueName: \"kubernetes.io/projected/db35bfba-1b72-4636-b79d-8985b3070ffe-kube-api-access-kcgkx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:31.286685 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.286673    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db35bfba-1b72-4636-b79d-8985b3070ffe-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:31.292343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.292313    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"]
Oct 31 07:44:31.387611 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.387576    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac1558a-f5r5q\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.387765 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.387746    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvhg\" (UniqueName: \"kubernetes.io/projected/63062387-4639-49f0-a3f7-6b6150b199ce-kube-api-access-dxvhg\") pod \"maas-api-revoke-sub-cac1558a-f5r5q\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.387861 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.387845    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac1558a-f5r5q\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.388377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.388351    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cac1558a-f5r5q\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.390203 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.390179    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cac1558a-f5r5q\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.407173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.407146    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvhg\" (UniqueName: \"kubernetes.io/projected/63062387-4639-49f0-a3f7-6b6150b199ce-kube-api-access-dxvhg\") pod \"maas-api-revoke-sub-cac1558a-f5r5q\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.603190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.603096    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:31.922094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.922053    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v" podUID="c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:44:31.931014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.930987    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"
Oct 31 07:44:31.931176 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.931014    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc" event={"ID":"db35bfba-1b72-4636-b79d-8985b3070ffe","Type":"ContainerDied","Data":"c85b7170e1193d8647d2b1949f61d9b0d1c0d4099d6fe773330fb4e468fca96e"}
Oct 31 07:44:31.931176 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.931082    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c85b7170e1193d8647d2b1949f61d9b0d1c0d4099d6fe773330fb4e468fca96e"
Oct 31 07:44:31.938895 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:31.938874    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"]
Oct 31 07:44:31.940915 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:31.940871    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63062387_4639_49f0_a3f7_6b6150b199ce.slice/crio-619fc65d5bd662d5e7eb871d6a3090fb2606b34bc62a6c9f7e3804ef9880c13e WatchSource:0}: Error finding container 619fc65d5bd662d5e7eb871d6a3090fb2606b34bc62a6c9f7e3804ef9880c13e: Status 404 returned error can't find the container with id 619fc65d5bd662d5e7eb871d6a3090fb2606b34bc62a6c9f7e3804ef9880c13e
Oct 31 07:44:32.910875 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:32.910836    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"]
Oct 31 07:44:32.942958 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:32.942915    5829 generic.go:356] "Generic (PLEG): container finished" podID="63062387-4639-49f0-a3f7-6b6150b199ce" containerID="873bf6b49e1edcc9391243e38826ae51f1afcaea6fabe98d19e275fd30f51e16" exitCode=0
Oct 31 07:44:32.943109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:32.943001    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q" event={"ID":"63062387-4639-49f0-a3f7-6b6150b199ce","Type":"ContainerDied","Data":"873bf6b49e1edcc9391243e38826ae51f1afcaea6fabe98d19e275fd30f51e16"}
Oct 31 07:44:32.943109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:32.943054    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q" event={"ID":"63062387-4639-49f0-a3f7-6b6150b199ce","Type":"ContainerStarted","Data":"619fc65d5bd662d5e7eb871d6a3090fb2606b34bc62a6c9f7e3804ef9880c13e"}
Oct 31 07:44:32.944769 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:32.944748    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"]
Oct 31 07:44:33.676152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.676109    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"]
Oct 31 07:44:33.682004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.681956    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.712535 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.712505    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"]
Oct 31 07:44:33.740036 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.739995    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-podinfo\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.740511 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.740479    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-envoy\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.755522 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.755475    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6229c3a0-d54b-49ac-9255-7c9d9594774a-istiod-ca-cert\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.755686 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.755545    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-credential-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.756224 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.755683    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-certs\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.756224 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.755712    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.756224 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.755831    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-data\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.756224 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.755895    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-token\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.756224 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.756033    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clph7\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-kube-api-access-clph7\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.857982 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.857940    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-envoy\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858050    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6229c3a0-d54b-49ac-9255-7c9d9594774a-istiod-ca-cert\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858080    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-credential-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858133    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-certs\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858152    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858358    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-data\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858420    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-token\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858489    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clph7\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-kube-api-access-clph7\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858566    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-podinfo\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858597    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-credential-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858661    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-socket\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858680    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-certs\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858875 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858844    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-data\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.858984 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.858965    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6229c3a0-d54b-49ac-9255-7c9d9594774a-istiod-ca-cert\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.860580 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.860556    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-envoy\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.860918 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.860900    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-podinfo\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.877982 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.877951    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-token\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:33.878122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:33.878103    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clph7\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-kube-api-access-clph7\") pod \"e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") " pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:34.011600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.011555    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:34.113324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.113265    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:34.165663 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.165617    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63062387-4639-49f0-a3f7-6b6150b199ce-kube-api-access-dxvhg\" (UniqueName: \"kubernetes.io/projected/63062387-4639-49f0-a3f7-6b6150b199ce-kube-api-access-dxvhg\") pod \"63062387-4639-49f0-a3f7-6b6150b199ce\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") "
Oct 31 07:44:34.165909 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.165878    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-openshift-ca\") pod \"63062387-4639-49f0-a3f7-6b6150b199ce\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") "
Oct 31 07:44:34.166061 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.165954    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-xks-ca\") pod \"63062387-4639-49f0-a3f7-6b6150b199ce\" (UID: \"63062387-4639-49f0-a3f7-6b6150b199ce\") "
Oct 31 07:44:34.166315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.166258    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-openshift-ca" pod "63062387-4639-49f0-a3f7-6b6150b199ce" (UID: "63062387-4639-49f0-a3f7-6b6150b199ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:34.166680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.166661    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:34.168463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.168433    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63062387-4639-49f0-a3f7-6b6150b199ce-kube-api-access-dxvhg" pod "63062387-4639-49f0-a3f7-6b6150b199ce" (UID: "63062387-4639-49f0-a3f7-6b6150b199ce"). InnerVolumeSpecName "kube-api-access-dxvhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:34.168627 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.168608    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-xks-ca" pod "63062387-4639-49f0-a3f7-6b6150b199ce" (UID: "63062387-4639-49f0-a3f7-6b6150b199ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:34.268272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.268189    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63062387-4639-49f0-a3f7-6b6150b199ce-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:34.268272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.268217    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxvhg\" (UniqueName: \"kubernetes.io/projected/63062387-4639-49f0-a3f7-6b6150b199ce-kube-api-access-dxvhg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:34.273170 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.273119    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"]
Oct 31 07:44:34.274394 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.274198    5829 cpu_manager.go:414] "containerMap: removing container" podUID="63062387-4639-49f0-a3f7-6b6150b199ce" containerName="revoke-keys"
Oct 31 07:44:34.274394 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.274223    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63062387-4639-49f0-a3f7-6b6150b199ce" containerName="revoke-keys"
Oct 31 07:44:34.274613 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.274526    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="63062387-4639-49f0-a3f7-6b6150b199ce" containerName="revoke-keys"
Oct 31 07:44:34.282274 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.282238    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.342660 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.342264    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"]
Oct 31 07:44:34.371411 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.371336    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td5wk\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-kube-api-access-td5wk\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.371594 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.371558    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-data\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.371709 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.371694    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-certs\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.371781 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.371764    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-envoy\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.371843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.371808    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-credential-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.371897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.371883    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-token\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.372002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.371987    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-podinfo\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.372054 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.372018    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istiod-ca-cert\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.372188 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.372108    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.377421 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:34.377379    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6229c3a0_d54b_49ac_9255_7c9d9594774a.slice/crio-baa789c5b64273d32cf6cbab698443b6d11cae66d5c0cc6724298b5e83813601 WatchSource:0}: Error finding container baa789c5b64273d32cf6cbab698443b6d11cae66d5c0cc6724298b5e83813601: Status 404 returned error can't find the container with id baa789c5b64273d32cf6cbab698443b6d11cae66d5c0cc6724298b5e83813601
Oct 31 07:44:34.377558 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.377455    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"]
Oct 31 07:44:34.381998 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.381950    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:34.382138 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.382050    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:34.382138 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.382101    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:34.473493 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.473437    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td5wk\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-kube-api-access-td5wk\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.473674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.473537    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-data\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.473674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.473596    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-certs\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.473674 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.473647    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-envoy\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.473844 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.473687    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-credential-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.473844 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.473732    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-token\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.473844 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.473827    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-podinfo\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.473993 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.473855    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istiod-ca-cert\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.473993 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.473932    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.474101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.474016    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-data\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.474594 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.474272    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-certs\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.474594 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.474369    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-credential-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.474594 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.474558    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-socket\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.474876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.474850    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istiod-ca-cert\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.476353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.476313    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-envoy\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.477016 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.476996    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-podinfo\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.493363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.493326    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-token\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.493508 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.493441    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td5wk\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-kube-api-access-td5wk\") pod \"e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") " pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.629911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.629825    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:34.964082 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.964024    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"]
Oct 31 07:44:34.975543 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:34.975489    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0974f3c_81c0_46f2_8500_bc7893afc3ea.slice/crio-75f86d002ab9a19f68d43b823865eb157e8e1e92470f620e9077080b3898ee85 WatchSource:0}: Error finding container 75f86d002ab9a19f68d43b823865eb157e8e1e92470f620e9077080b3898ee85: Status 404 returned error can't find the container with id 75f86d002ab9a19f68d43b823865eb157e8e1e92470f620e9077080b3898ee85
Oct 31 07:44:34.978707 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.978663    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:34.978823 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.978756    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:34.978885 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.978856    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:34.979549 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.979515    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" event={"ID":"6229c3a0-d54b-49ac-9255-7c9d9594774a","Type":"ContainerStarted","Data":"92e9636f6fade172d695b32ffbcb6f7fff3d33f19df4979c01c2b6a78bce757d"}
Oct 31 07:44:34.979649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.979580    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" event={"ID":"6229c3a0-d54b-49ac-9255-7c9d9594774a","Type":"ContainerStarted","Data":"baa789c5b64273d32cf6cbab698443b6d11cae66d5c0cc6724298b5e83813601"}
Oct 31 07:44:34.982821 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.982802    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"
Oct 31 07:44:34.982911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.982800    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q" event={"ID":"63062387-4639-49f0-a3f7-6b6150b199ce","Type":"ContainerDied","Data":"619fc65d5bd662d5e7eb871d6a3090fb2606b34bc62a6c9f7e3804ef9880c13e"}
Oct 31 07:44:34.982964 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:34.982924    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="619fc65d5bd662d5e7eb871d6a3090fb2606b34bc62a6c9f7e3804ef9880c13e"
Oct 31 07:44:35.016311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:35.014596    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:35.016311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:35.014779    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 31 07:44:35.016311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:35.014825    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 31 07:44:35.035900 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:35.035193    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podStartSLOduration=2.03517287 podStartE2EDuration="2.03517287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:35.024323475 +0000 UTC m=+939.163484064" watchObservedRunningTime="2026-10-31 07:44:35.03517287 +0000 UTC m=+939.174333459"
Oct 31 07:44:36.000586 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.000494    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" event={"ID":"d0974f3c-81c0-46f2-8500-bc7893afc3ea","Type":"ContainerStarted","Data":"276b1fb5db439d4297b4dd224cad6d237ef8077326a1beb3cbab9652cf55316b"}
Oct 31 07:44:36.000586 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.000554    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" event={"ID":"d0974f3c-81c0-46f2-8500-bc7893afc3ea","Type":"ContainerStarted","Data":"75f86d002ab9a19f68d43b823865eb157e8e1e92470f620e9077080b3898ee85"}
Oct 31 07:44:36.028307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.025526    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 31 07:44:36.028307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.025586    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 31 07:44:36.072783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.072717    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" podStartSLOduration=2.07270161 podStartE2EDuration="2.07270161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:36.055827323 +0000 UTC m=+940.194987912" watchObservedRunningTime="2026-10-31 07:44:36.07270161 +0000 UTC m=+940.211862198"
Oct 31 07:44:36.177310 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.176255    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"]
Oct 31 07:44:36.182647 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.182486    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.183617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.183583    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"]
Oct 31 07:44:36.198498 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.198472    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:44:36.198717 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.198530    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:44:36.198846 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.198703    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:44:36.199165 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.199141    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:44:36.353368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.346004    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09760dd-7ltcl\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.353368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.346210    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbq88\" (UniqueName: \"kubernetes.io/projected/98d95c9d-2384-4cba-a30c-4823af7b31d1-kube-api-access-xbq88\") pod \"maas-api-revoke-sub-f09760dd-7ltcl\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.353368 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.346251    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09760dd-7ltcl\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.517558 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.517509    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09760dd-7ltcl\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.527857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.527824    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbq88\" (UniqueName: \"kubernetes.io/projected/98d95c9d-2384-4cba-a30c-4823af7b31d1-kube-api-access-xbq88\") pod \"maas-api-revoke-sub-f09760dd-7ltcl\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.528188 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.528165    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09760dd-7ltcl\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.544933 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.544898    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f09760dd-7ltcl\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.558344 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.545708    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f09760dd-7ltcl\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.590464 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.590253    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbq88\" (UniqueName: \"kubernetes.io/projected/98d95c9d-2384-4cba-a30c-4823af7b31d1-kube-api-access-xbq88\") pod \"maas-api-revoke-sub-f09760dd-7ltcl\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.643179 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.643137    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:36.666223 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.665994    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 07:44:36.666223 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.666067    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 07:44:36.765308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.763763    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"]
Oct 31 07:44:36.781309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.781250    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:36.789105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.789051    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"]
Oct 31 07:44:36.822454 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.811805    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:36.890895 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.881763    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f64sn\" (UniqueName: \"kubernetes.io/projected/330b31f3-9f9f-4e09-8c5e-fa542b87c526-kube-api-access-f64sn\") pod \"maas-api-revoke-sub-6f7bf75b-nwmh2\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:36.890895 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.881885    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f7bf75b-nwmh2\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:36.890895 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.881987    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f7bf75b-nwmh2\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:36.973261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.973225    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"]
Oct 31 07:44:36.985111 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.985061    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64sn\" (UniqueName: \"kubernetes.io/projected/330b31f3-9f9f-4e09-8c5e-fa542b87c526-kube-api-access-f64sn\") pod \"maas-api-revoke-sub-6f7bf75b-nwmh2\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:36.985313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.985180    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f7bf75b-nwmh2\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:36.985313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.985266    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f7bf75b-nwmh2\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:36.986248 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.986218    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f7bf75b-nwmh2\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:36.987168 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:36.987143    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.002440 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.002409    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f7bf75b-nwmh2\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:37.002707 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.002670    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 07:44:37.002805 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.002722    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 07:44:37.002863 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.002845    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jbt7z\""
Oct 31 07:44:37.069584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.065216    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 07:44:37.069584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.067683    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 31 07:44:37.069584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.067736    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 31 07:44:37.077101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.077070    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"]
Oct 31 07:44:37.077546 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.077523    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f64sn\" (UniqueName: \"kubernetes.io/projected/330b31f3-9f9f-4e09-8c5e-fa542b87c526-kube-api-access-f64sn\") pod \"maas-api-revoke-sub-6f7bf75b-nwmh2\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:37.091817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.091785    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f012873-83b7-487e-8da9-04fa08087316-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.091963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.091831    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.091963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.091899    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsk97\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-kube-api-access-wsk97\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.092085 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.092025    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.092141 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.092127    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.092210 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.092198    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.092333 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.092234    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.184607 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.184528    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:37.194776 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.194745    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.194913 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.194851    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.194964 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.194953    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.195006 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.194994    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.195059 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.195049    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f012873-83b7-487e-8da9-04fa08087316-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.195095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.195068    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.195127 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.195110    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsk97\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-kube-api-access-wsk97\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.198555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.198526    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f012873-83b7-487e-8da9-04fa08087316-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.198707 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.198632    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.198794 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.198539    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.199194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.199166    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.199536 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.199516    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.273429 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.270519    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.277760 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.276295    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsk97\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-kube-api-access-wsk97\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g8928\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.393257 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.393229    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:37.395538 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.394877    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"]
Oct 31 07:44:37.666997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.666950    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"]
Oct 31 07:44:37.667517 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.667484    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 07:44:37.667726 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.667698    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 07:44:37.689570 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:37.689526    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod330b31f3_9f9f_4e09_8c5e_fa542b87c526.slice/crio-77f881a919d58e832275ccff2d76b9c725c88b9c304b1f5fee7276e94599d94a WatchSource:0}: Error finding container 77f881a919d58e832275ccff2d76b9c725c88b9c304b1f5fee7276e94599d94a: Status 404 returned error can't find the container with id 77f881a919d58e832275ccff2d76b9c725c88b9c304b1f5fee7276e94599d94a
Oct 31 07:44:37.882347 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:37.882316    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f012873_83b7_487e_8da9_04fa08087316.slice/crio-9c4fb309fd722cb89a1e2ae5be13e461bdb57a2da917c1b729334f41b3790d97 WatchSource:0}: Error finding container 9c4fb309fd722cb89a1e2ae5be13e461bdb57a2da917c1b729334f41b3790d97: Status 404 returned error can't find the container with id 9c4fb309fd722cb89a1e2ae5be13e461bdb57a2da917c1b729334f41b3790d97
Oct 31 07:44:37.883370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.883348    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"]
Oct 31 07:44:37.885501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.885480    5829 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 07:44:37.940190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.939636    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2" podUID="53782e9d-d7b0-4a17-b756-f14960c29c2c" containerName="payload-processing" containerID="cri-o://7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76" gracePeriod=25
Oct 31 07:44:37.959539 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:37.959341    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v" podUID="6e369c6a-79e7-4aed-a1c3-a756e512372c" containerName="payload-pre-processing" containerID="cri-o://4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac" gracePeriod=25
Oct 31 07:44:38.034378 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.029564    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 31 07:44:38.034378 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.029631    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 31 07:44:38.146631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.146586    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"]
Oct 31 07:44:38.161407 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.161368    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl" event={"ID":"98d95c9d-2384-4cba-a30c-4823af7b31d1","Type":"ContainerStarted","Data":"798785908f6bac78b85067be2c7efc657cb0772092febe5cd39a8fdd1dc73c4f"}
Oct 31 07:44:38.161708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.161683    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl" event={"ID":"98d95c9d-2384-4cba-a30c-4823af7b31d1","Type":"ContainerStarted","Data":"60d397e0cd97a9ae3a22c0563671b9a7b11a87cb22b73aba4bb69e94efd1090c"}
Oct 31 07:44:38.162952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.162918    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-648fbe-xn97x"]
Oct 31 07:44:38.163308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.163274    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928" event={"ID":"7f012873-83b7-487e-8da9-04fa08087316","Type":"ContainerStarted","Data":"9c4fb309fd722cb89a1e2ae5be13e461bdb57a2da917c1b729334f41b3790d97"}
Oct 31 07:44:38.164657 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.164635    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2" event={"ID":"330b31f3-9f9f-4e09-8c5e-fa542b87c526","Type":"ContainerStarted","Data":"77f881a919d58e832275ccff2d76b9c725c88b9c304b1f5fee7276e94599d94a"}
Oct 31 07:44:38.201654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.201610    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl" podStartSLOduration=2.201596593 podStartE2EDuration="2.201596593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:38.200713716 +0000 UTC m=+942.339874303" watchObservedRunningTime="2026-10-31 07:44:38.201596593 +0000 UTC m=+942.340757216"
Oct 31 07:44:38.631802 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.631760    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 07:44:38.632095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:38.632058    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 07:44:39.013190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:39.012714    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 31 07:44:39.013190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:39.012781    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 31 07:44:39.177398 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:39.177310    5829 generic.go:356] "Generic (PLEG): container finished" podID="98d95c9d-2384-4cba-a30c-4823af7b31d1" containerID="798785908f6bac78b85067be2c7efc657cb0772092febe5cd39a8fdd1dc73c4f" exitCode=0
Oct 31 07:44:39.177876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:39.177399    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl" event={"ID":"98d95c9d-2384-4cba-a30c-4823af7b31d1","Type":"ContainerDied","Data":"798785908f6bac78b85067be2c7efc657cb0772092febe5cd39a8fdd1dc73c4f"}
Oct 31 07:44:39.179937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:39.179911    5829 generic.go:356] "Generic (PLEG): container finished" podID="330b31f3-9f9f-4e09-8c5e-fa542b87c526" containerID="e3884fc42ccbf918be57783c50c7de5aec6b6048e425886398e6295292b8f21a" exitCode=0
Oct 31 07:44:39.180082 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:39.179944    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2" event={"ID":"330b31f3-9f9f-4e09-8c5e-fa542b87c526","Type":"ContainerDied","Data":"e3884fc42ccbf918be57783c50c7de5aec6b6048e425886398e6295292b8f21a"}
Oct 31 07:44:39.632312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:39.631747    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 07:44:39.632312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:39.631835    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 07:44:39.979253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:39.979215    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41dcd454-0846-4f21-a3dd-fe4dc26e26ef" path="/var/lib/kubelet/pods/41dcd454-0846-4f21-a3dd-fe4dc26e26ef/volumes"
Oct 31 07:44:40.037321 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.029438    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 31 07:44:40.037321 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.029530    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 31 07:44:40.588314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.587854    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:40.656470 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.656416    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 07:44:40.656654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.656491    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 07:44:40.669826 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.669796    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:40.679570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.677350    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/330b31f3-9f9f-4e09-8c5e-fa542b87c526-kube-api-access-f64sn\" (UniqueName: \"kubernetes.io/projected/330b31f3-9f9f-4e09-8c5e-fa542b87c526-kube-api-access-f64sn\") pod \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") "
Oct 31 07:44:40.679570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.677513    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d95c9d-2384-4cba-a30c-4823af7b31d1-kube-api-access-xbq88\" (UniqueName: \"kubernetes.io/projected/98d95c9d-2384-4cba-a30c-4823af7b31d1-kube-api-access-xbq88\") pod \"98d95c9d-2384-4cba-a30c-4823af7b31d1\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") "
Oct 31 07:44:40.679570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.677543    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-xks-ca\") pod \"98d95c9d-2384-4cba-a30c-4823af7b31d1\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") "
Oct 31 07:44:40.679570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.677604    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-openshift-ca\") pod \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") "
Oct 31 07:44:40.679570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.677622    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-xks-ca\") pod \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\" (UID: \"330b31f3-9f9f-4e09-8c5e-fa542b87c526\") "
Oct 31 07:44:40.679570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.677679    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-openshift-ca\") pod \"98d95c9d-2384-4cba-a30c-4823af7b31d1\" (UID: \"98d95c9d-2384-4cba-a30c-4823af7b31d1\") "
Oct 31 07:44:40.679570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.679241    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-openshift-ca" pod "330b31f3-9f9f-4e09-8c5e-fa542b87c526" (UID: "330b31f3-9f9f-4e09-8c5e-fa542b87c526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:40.680086 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.680054    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-openshift-ca" pod "98d95c9d-2384-4cba-a30c-4823af7b31d1" (UID: "98d95c9d-2384-4cba-a30c-4823af7b31d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:40.690842 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.690712    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-xks-ca" pod "330b31f3-9f9f-4e09-8c5e-fa542b87c526" (UID: "330b31f3-9f9f-4e09-8c5e-fa542b87c526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:40.690842 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.690796    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d95c9d-2384-4cba-a30c-4823af7b31d1-kube-api-access-xbq88" pod "98d95c9d-2384-4cba-a30c-4823af7b31d1" (UID: "98d95c9d-2384-4cba-a30c-4823af7b31d1"). InnerVolumeSpecName "kube-api-access-xbq88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:40.693655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.693611    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-xks-ca" pod "98d95c9d-2384-4cba-a30c-4823af7b31d1" (UID: "98d95c9d-2384-4cba-a30c-4823af7b31d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:40.735754 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.735692    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330b31f3-9f9f-4e09-8c5e-fa542b87c526-kube-api-access-f64sn" pod "330b31f3-9f9f-4e09-8c5e-fa542b87c526" (UID: "330b31f3-9f9f-4e09-8c5e-fa542b87c526"). InnerVolumeSpecName "kube-api-access-f64sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:40.779369 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.779332    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbq88\" (UniqueName: \"kubernetes.io/projected/98d95c9d-2384-4cba-a30c-4823af7b31d1-kube-api-access-xbq88\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:40.779369 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.779371    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:40.779570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.779386    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:40.779570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.779395    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/330b31f3-9f9f-4e09-8c5e-fa542b87c526-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:40.779570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.779404    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98d95c9d-2384-4cba-a30c-4823af7b31d1-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:40.779570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.779415    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f64sn\" (UniqueName: \"kubernetes.io/projected/330b31f3-9f9f-4e09-8c5e-fa542b87c526-kube-api-access-f64sn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:40.828306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.803790    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"]
Oct 31 07:44:40.828306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.810106    5829 cpu_manager.go:414] "containerMap: removing container" podUID="98d95c9d-2384-4cba-a30c-4823af7b31d1" containerName="revoke-keys"
Oct 31 07:44:40.828306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.810134    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d95c9d-2384-4cba-a30c-4823af7b31d1" containerName="revoke-keys"
Oct 31 07:44:40.828306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.810166    5829 cpu_manager.go:414] "containerMap: removing container" podUID="330b31f3-9f9f-4e09-8c5e-fa542b87c526" containerName="revoke-keys"
Oct 31 07:44:40.828306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.810174    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="330b31f3-9f9f-4e09-8c5e-fa542b87c526" containerName="revoke-keys"
Oct 31 07:44:40.828306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.810459    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d95c9d-2384-4cba-a30c-4823af7b31d1" containerName="revoke-keys"
Oct 31 07:44:40.828306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.810483    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="330b31f3-9f9f-4e09-8c5e-fa542b87c526" containerName="revoke-keys"
Oct 31 07:44:40.828306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.819930    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:40.848001 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.847969    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"]
Oct 31 07:44:40.880827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.880791    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b0f715-kqgr6\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:40.881891 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.880994    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tkvw\" (UniqueName: \"kubernetes.io/projected/42b43209-4f04-4826-9f20-a86825f26d63-kube-api-access-6tkvw\") pod \"maas-api-revoke-sub-c2b0f715-kqgr6\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:40.881891 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.881045    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b0f715-kqgr6\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:40.985375 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.983809    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tkvw\" (UniqueName: \"kubernetes.io/projected/42b43209-4f04-4826-9f20-a86825f26d63-kube-api-access-6tkvw\") pod \"maas-api-revoke-sub-c2b0f715-kqgr6\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:40.985375 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.983873    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b0f715-kqgr6\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:40.985375 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.984041    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b0f715-kqgr6\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:40.985649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.985421    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2b0f715-kqgr6\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:40.988247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:40.988225    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2b0f715-kqgr6\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:41.014302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.013642    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 31 07:44:41.014302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.013702    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 31 07:44:41.026304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.019942    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tkvw\" (UniqueName: \"kubernetes.io/projected/42b43209-4f04-4826-9f20-a86825f26d63-kube-api-access-6tkvw\") pod \"maas-api-revoke-sub-c2b0f715-kqgr6\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:41.157059 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.157021    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:41.228600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.228554    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl" event={"ID":"98d95c9d-2384-4cba-a30c-4823af7b31d1","Type":"ContainerDied","Data":"60d397e0cd97a9ae3a22c0563671b9a7b11a87cb22b73aba4bb69e94efd1090c"}
Oct 31 07:44:41.228761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.228625    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60d397e0cd97a9ae3a22c0563671b9a7b11a87cb22b73aba4bb69e94efd1090c"
Oct 31 07:44:41.228761 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.228706    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"
Oct 31 07:44:41.244704 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.244612    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2" event={"ID":"330b31f3-9f9f-4e09-8c5e-fa542b87c526","Type":"ContainerDied","Data":"77f881a919d58e832275ccff2d76b9c725c88b9c304b1f5fee7276e94599d94a"}
Oct 31 07:44:41.244963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.244946    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77f881a919d58e832275ccff2d76b9c725c88b9c304b1f5fee7276e94599d94a"
Oct 31 07:44:41.245119 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.245108    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"
Oct 31 07:44:41.491617 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:41.491579    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b43209_4f04_4826_9f20_a86825f26d63.slice/crio-edf22e92fb42b091924baf66ded27ef1b7fc75377938285d705d0539a696ea28 WatchSource:0}: Error finding container edf22e92fb42b091924baf66ded27ef1b7fc75377938285d705d0539a696ea28: Status 404 returned error can't find the container with id edf22e92fb42b091924baf66ded27ef1b7fc75377938285d705d0539a696ea28
Oct 31 07:44:41.494233 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.494203    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"]
Oct 31 07:44:41.630706 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.630669    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 31 07:44:41.631132 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.630735    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 31 07:44:41.922304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:41.922237    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v" podUID="c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:44:42.096301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.095821    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"]
Oct 31 07:44:42.164305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.163679    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"]
Oct 31 07:44:42.164305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.163841    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.219453 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.217376    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" containerID="cri-o://819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846" gracePeriod=45
Oct 31 07:44:42.219453 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.217552    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.219453 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.217668    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.219453 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.217728    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckspt\" (UniqueName: \"kubernetes.io/projected/90f033de-bdf1-430f-94cc-d1fd0b452c9d-kube-api-access-ckspt\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.242261 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.242209    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.94:8000/ready\": dial tcp 10.134.0.94:8000: connect: connection refused"
Oct 31 07:44:42.269614 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.267157    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6" event={"ID":"42b43209-4f04-4826-9f20-a86825f26d63","Type":"ContainerStarted","Data":"9fc40a611c69704a8cb1d9570c1ede3ab331a9d0db0fe81a2d6d730220a2143c"}
Oct 31 07:44:42.269614 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.267219    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6" event={"ID":"42b43209-4f04-4826-9f20-a86825f26d63","Type":"ContainerStarted","Data":"edf22e92fb42b091924baf66ded27ef1b7fc75377938285d705d0539a696ea28"}
Oct 31 07:44:42.314316 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.312780    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6" podStartSLOduration=2.312759506 podStartE2EDuration="2.312759506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:42.307247646 +0000 UTC m=+946.446408234" watchObservedRunningTime="2026-10-31 07:44:42.312759506 +0000 UTC m=+946.451920097"
Oct 31 07:44:42.324302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.319636    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:42.324302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.321060    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.324302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.321114    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.324302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.321155    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckspt\" (UniqueName: \"kubernetes.io/projected/90f033de-bdf1-430f-94cc-d1fd0b452c9d-kube-api-access-ckspt\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.324302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.321680    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:42.324302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.322404    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.329301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.324994    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:44:42.329301 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.327354    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.343898 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.343860    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckspt\" (UniqueName: \"kubernetes.io/projected/90f033de-bdf1-430f-94cc-d1fd0b452c9d-kube-api-access-ckspt\") pod \"maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.547868 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.518638    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:42.653433 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.653394    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:44:42.738539 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.738349    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-home" pod "a158b083-77df-454a-af0d-e919c4b320c0" (UID: "a158b083-77df-454a-af0d-e919c4b320c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:42.738945 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.738839    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-home\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-home\") pod \"a158b083-77df-454a-af0d-e919c4b320c0\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") "
Oct 31 07:44:42.739244 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.739222    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a158b083-77df-454a-af0d-e919c4b320c0-kube-api-access-jslbz\" (UniqueName: \"kubernetes.io/projected/a158b083-77df-454a-af0d-e919c4b320c0-kube-api-access-jslbz\") pod \"a158b083-77df-454a-af0d-e919c4b320c0\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") "
Oct 31 07:44:42.740643 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.740623    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-dshm\") pod \"a158b083-77df-454a-af0d-e919c4b320c0\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") "
Oct 31 07:44:42.741275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.741256    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a158b083-77df-454a-af0d-e919c4b320c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a158b083-77df-454a-af0d-e919c4b320c0-tls-certs\") pod \"a158b083-77df-454a-af0d-e919c4b320c0\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") "
Oct 31 07:44:42.742406 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.742370    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-tmp-dir\") pod \"a158b083-77df-454a-af0d-e919c4b320c0\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") "
Oct 31 07:44:42.742856 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.742521    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-tmp-dir" pod "a158b083-77df-454a-af0d-e919c4b320c0" (UID: "a158b083-77df-454a-af0d-e919c4b320c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:42.742976 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.742950    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-model-cache\") pod \"a158b083-77df-454a-af0d-e919c4b320c0\" (UID: \"a158b083-77df-454a-af0d-e919c4b320c0\") "
Oct 31 07:44:42.745566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.744129    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:42.745566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.744158    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:42.745566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.745370    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-model-cache" pod "a158b083-77df-454a-af0d-e919c4b320c0" (UID: "a158b083-77df-454a-af0d-e919c4b320c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:42.759259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.759085    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-dshm" pod "a158b083-77df-454a-af0d-e919c4b320c0" (UID: "a158b083-77df-454a-af0d-e919c4b320c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:42.759259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.759209    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a158b083-77df-454a-af0d-e919c4b320c0-kube-api-access-jslbz" pod "a158b083-77df-454a-af0d-e919c4b320c0" (UID: "a158b083-77df-454a-af0d-e919c4b320c0"). InnerVolumeSpecName "kube-api-access-jslbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:42.769586 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.769531    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a158b083-77df-454a-af0d-e919c4b320c0-tls-certs" pod "a158b083-77df-454a-af0d-e919c4b320c0" (UID: "a158b083-77df-454a-af0d-e919c4b320c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:42.845903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.845810    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:42.845903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.845846    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jslbz\" (UniqueName: \"kubernetes.io/projected/a158b083-77df-454a-af0d-e919c4b320c0-kube-api-access-jslbz\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:42.845903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.845861    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a158b083-77df-454a-af0d-e919c4b320c0-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:42.845903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.845876    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a158b083-77df-454a-af0d-e919c4b320c0-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:42.896511 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.896476    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:44:42.915302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.914568    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"]
Oct 31 07:44:42.948817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.948776    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-plugins-config-volume\") pod \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\" (UID: \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\") "
Oct 31 07:44:42.949174 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.949145    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-plugins-config-volume" pod "c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" (UID: "c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:42.949326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.949306    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-kube-api-access-g2s2d\" (UniqueName: \"kubernetes.io/projected/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-kube-api-access-g2s2d\") pod \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\" (UID: \"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8\") "
Oct 31 07:44:42.950118 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.950095    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:42.958448 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.958393    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-kube-api-access-g2s2d" pod "c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" (UID: "c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8"). InnerVolumeSpecName "kube-api-access-g2s2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:42.958829 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.958799    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:42.959913 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.959766    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:42.964162 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:42.964142    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:44:43.004372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.004299    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:43.004607 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.004574    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:44:43.019476 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:43.019431    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f033de_bdf1_430f_94cc_d1fd0b452c9d.slice/crio-85107410244cdd130e1ade696106b889effa11d886d8df3cd132ba643fb8b8ef WatchSource:0}: Error finding container 85107410244cdd130e1ade696106b889effa11d886d8df3cd132ba643fb8b8ef: Status 404 returned error can't find the container with id 85107410244cdd130e1ade696106b889effa11d886d8df3cd132ba643fb8b8ef
Oct 31 07:44:43.052584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.052552    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2s2d\" (UniqueName: \"kubernetes.io/projected/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8-kube-api-access-g2s2d\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:43.280618 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.280523    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928" event={"ID":"7f012873-83b7-487e-8da9-04fa08087316","Type":"ContainerStarted","Data":"f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972"}
Oct 31 07:44:43.280787 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.280636    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:43.282659 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.282630    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5" event={"ID":"90f033de-bdf1-430f-94cc-d1fd0b452c9d","Type":"ContainerStarted","Data":"85107410244cdd130e1ade696106b889effa11d886d8df3cd132ba643fb8b8ef"}
Oct 31 07:44:43.284017 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.283996    5829 generic.go:356] "Generic (PLEG): container finished" podID="a158b083-77df-454a-af0d-e919c4b320c0" containerID="819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846" exitCode=0
Oct 31 07:44:43.284133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.284040    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"
Oct 31 07:44:43.284133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.284077    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" event={"ID":"a158b083-77df-454a-af0d-e919c4b320c0","Type":"ContainerDied","Data":"819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846"}
Oct 31 07:44:43.284133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.284111    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk" event={"ID":"a158b083-77df-454a-af0d-e919c4b320c0","Type":"ContainerDied","Data":"15586bf617bb7766732dbcc4e7fe9718d04b48459459e7f51b58f82a62828f62"}
Oct 31 07:44:43.284326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.284130    5829 scope.go:122] "RemoveContainer" containerID="819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846"
Oct 31 07:44:43.286010 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.285987    5829 generic.go:356] "Generic (PLEG): container finished" podID="42b43209-4f04-4826-9f20-a86825f26d63" containerID="9fc40a611c69704a8cb1d9570c1ede3ab331a9d0db0fe81a2d6d730220a2143c" exitCode=0
Oct 31 07:44:43.286122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.286067    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6" event={"ID":"42b43209-4f04-4826-9f20-a86825f26d63","Type":"ContainerDied","Data":"9fc40a611c69704a8cb1d9570c1ede3ab331a9d0db0fe81a2d6d730220a2143c"}
Oct 31 07:44:43.287694 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.287669    5829 generic.go:356] "Generic (PLEG): container finished" podID="c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" containerID="f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80" exitCode=0
Oct 31 07:44:43.287800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.287726    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"
Oct 31 07:44:43.287857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.287726    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v" event={"ID":"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8","Type":"ContainerDied","Data":"f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80"}
Oct 31 07:44:43.287910 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.287861    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v" event={"ID":"c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8","Type":"ContainerDied","Data":"13fe9412c71acc8a0fff0fed1edb123f0ce88449643d0ffccbdbe0eb5cb7d4fd"}
Oct 31 07:44:43.298979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.298953    5829 scope.go:122] "RemoveContainer" containerID="819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846"
Oct 31 07:44:43.299320 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:43.299269    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846\": container with ID starting with 819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846 not found: ID does not exist" containerID="819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846"
Oct 31 07:44:43.299395 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.299325    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846"} err="failed to get container status \"819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846\": rpc error: code = NotFound desc = could not find container \"819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846\": container with ID starting with 819aba9ae2516fe299057212003a2a6a6891aca2eb5572ffedbeadb40f8e4846 not found: ID does not exist"
Oct 31 07:44:43.299395 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.299353    5829 scope.go:122] "RemoveContainer" containerID="f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80"
Oct 31 07:44:43.312265 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.312245    5829 scope.go:122] "RemoveContainer" containerID="f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80"
Oct 31 07:44:43.312616 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:43.312595    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80\": container with ID starting with f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80 not found: ID does not exist" containerID="f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80"
Oct 31 07:44:43.312682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.312631    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80"} err="failed to get container status \"f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80\": rpc error: code = NotFound desc = could not find container \"f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80\": container with ID starting with f04d6ba4390313169ea129d9b133ebe4a1a3bb0496e45919cf6ed5a5c3b69a80 not found: ID does not exist"
Oct 31 07:44:43.328348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.328300    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928" podStartSLOduration=2.21120649 podStartE2EDuration="7.328274707s" totalImagesPullingTime="5.117068217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:43.324199103 +0000 UTC m=+947.463359722" watchObservedRunningTime="2026-10-31 07:44:43.328274707 +0000 UTC m=+947.467435296"
Oct 31 07:44:43.405783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.405746    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"]
Oct 31 07:44:43.413754 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.413722    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-027d2a/distinct-2-w6-027d2a-kserve-69ddcbb56d-t6jqk"]
Oct 31 07:44:43.440072 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.440045    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"]
Oct 31 07:44:43.444989 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.444969    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-648fbe-5546559b6c-hd58v"]
Oct 31 07:44:43.977160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.977125    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a158b083-77df-454a-af0d-e919c4b320c0" path="/var/lib/kubelet/pods/a158b083-77df-454a-af0d-e919c4b320c0/volumes"
Oct 31 07:44:43.977758 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:43.977739    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" path="/var/lib/kubelet/pods/c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8/volumes"
Oct 31 07:44:44.310890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.310789    5829 generic.go:356] "Generic (PLEG): container finished" podID="90f033de-bdf1-430f-94cc-d1fd0b452c9d" containerID="26424ce028309f1637ae62ba161cad2e7c9b3362b3a5b84f2da3365cdbabf004" exitCode=0
Oct 31 07:44:44.311488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.311046    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5" event={"ID":"90f033de-bdf1-430f-94cc-d1fd0b452c9d","Type":"ContainerDied","Data":"26424ce028309f1637ae62ba161cad2e7c9b3362b3a5b84f2da3365cdbabf004"}
Oct 31 07:44:44.316574 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.316549    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:44:44.501141 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.501113    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:44.596247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.596175    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b43209-4f04-4826-9f20-a86825f26d63-kube-api-access-6tkvw\" (UniqueName: \"kubernetes.io/projected/42b43209-4f04-4826-9f20-a86825f26d63-kube-api-access-6tkvw\") pod \"42b43209-4f04-4826-9f20-a86825f26d63\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") "
Oct 31 07:44:44.596396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.596327    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-xks-ca\") pod \"42b43209-4f04-4826-9f20-a86825f26d63\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") "
Oct 31 07:44:44.596438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.596396    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-openshift-ca\") pod \"42b43209-4f04-4826-9f20-a86825f26d63\" (UID: \"42b43209-4f04-4826-9f20-a86825f26d63\") "
Oct 31 07:44:44.597035 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.596994    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-openshift-ca" pod "42b43209-4f04-4826-9f20-a86825f26d63" (UID: "42b43209-4f04-4826-9f20-a86825f26d63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:44.597182 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.597158    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:44.598784 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.598756    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-xks-ca" pod "42b43209-4f04-4826-9f20-a86825f26d63" (UID: "42b43209-4f04-4826-9f20-a86825f26d63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:44.598880 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.598806    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b43209-4f04-4826-9f20-a86825f26d63-kube-api-access-6tkvw" pod "42b43209-4f04-4826-9f20-a86825f26d63" (UID: "42b43209-4f04-4826-9f20-a86825f26d63"). InnerVolumeSpecName "kube-api-access-6tkvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:44.699319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.699269    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tkvw\" (UniqueName: \"kubernetes.io/projected/42b43209-4f04-4826-9f20-a86825f26d63-kube-api-access-6tkvw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:44.699319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.699319    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b43209-4f04-4826-9f20-a86825f26d63-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:44.867193 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.867094    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v" podUID="6e369c6a-79e7-4aed-a1c3-a756e512372c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:44:44.867193 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.867151    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2" podUID="53782e9d-d7b0-4a17-b756-f14960c29c2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:44:44.924522 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.924477    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"]
Oct 31 07:44:44.932688 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.932409    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerName="istio-proxy" containerID="cri-o://8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62" gracePeriod=30
Oct 31 07:44:44.938730 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.938691    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"]
Oct 31 07:44:44.939303 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.938968    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy" containerID="cri-o://949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f" gracePeriod=30
Oct 31 07:44:44.945506 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.945478    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"]
Oct 31 07:44:44.945722 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:44.945698    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy" containerID="cri-o://2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01" gracePeriod=30
Oct 31 07:44:45.328029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.327995    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"
Oct 31 07:44:45.328029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.328026    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6" event={"ID":"42b43209-4f04-4826-9f20-a86825f26d63","Type":"ContainerDied","Data":"edf22e92fb42b091924baf66ded27ef1b7fc75377938285d705d0539a696ea28"}
Oct 31 07:44:45.328714 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.328091    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf22e92fb42b091924baf66ded27ef1b7fc75377938285d705d0539a696ea28"
Oct 31 07:44:45.519498 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.519472    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:45.619622 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.619533    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-xks-ca\") pod \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") "
Oct 31 07:44:45.619790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.619664    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f033de-bdf1-430f-94cc-d1fd0b452c9d-kube-api-access-ckspt\" (UniqueName: \"kubernetes.io/projected/90f033de-bdf1-430f-94cc-d1fd0b452c9d-kube-api-access-ckspt\") pod \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") "
Oct 31 07:44:45.619839 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.619793    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-openshift-ca\") pod \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\" (UID: \"90f033de-bdf1-430f-94cc-d1fd0b452c9d\") "
Oct 31 07:44:45.620702 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.620670    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-openshift-ca" pod "90f033de-bdf1-430f-94cc-d1fd0b452c9d" (UID: "90f033de-bdf1-430f-94cc-d1fd0b452c9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:45.622070 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.622038    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f033de-bdf1-430f-94cc-d1fd0b452c9d-kube-api-access-ckspt" pod "90f033de-bdf1-430f-94cc-d1fd0b452c9d" (UID: "90f033de-bdf1-430f-94cc-d1fd0b452c9d"). InnerVolumeSpecName "kube-api-access-ckspt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:45.622217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.622094    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-xks-ca" pod "90f033de-bdf1-430f-94cc-d1fd0b452c9d" (UID: "90f033de-bdf1-430f-94cc-d1fd0b452c9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:45.722809 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.722774    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:45.722809 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.722806    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f033de-bdf1-430f-94cc-d1fd0b452c9d-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:45.722809 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:45.722815    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckspt\" (UniqueName: \"kubernetes.io/projected/90f033de-bdf1-430f-94cc-d1fd0b452c9d-kube-api-access-ckspt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:46.338819 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:46.338735    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"
Oct 31 07:44:46.338819 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:46.338770    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5" event={"ID":"90f033de-bdf1-430f-94cc-d1fd0b452c9d","Type":"ContainerDied","Data":"85107410244cdd130e1ade696106b889effa11d886d8df3cd132ba643fb8b8ef"}
Oct 31 07:44:46.338819 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:46.338820    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85107410244cdd130e1ade696106b889effa11d886d8df3cd132ba643fb8b8ef"
Oct 31 07:44:46.519196 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:46.519150    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:44:46.716401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:46.716368    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"]
Oct 31 07:44:46.728783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:46.728736    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"]
Oct 31 07:44:47.960271 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.960235    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"]
Oct 31 07:44:47.961422 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961396    5829 cpu_manager.go:414] "containerMap: removing container" podUID="42b43209-4f04-4826-9f20-a86825f26d63" containerName="revoke-keys"
Oct 31 07:44:47.961422 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961422    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b43209-4f04-4826-9f20-a86825f26d63" containerName="revoke-keys"
Oct 31 07:44:47.961596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961442    5829 cpu_manager.go:414] "containerMap: removing container" podUID="90f033de-bdf1-430f-94cc-d1fd0b452c9d" containerName="revoke-keys"
Oct 31 07:44:47.961596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961451    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f033de-bdf1-430f-94cc-d1fd0b452c9d" containerName="revoke-keys"
Oct 31 07:44:47.961596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961505    5829 cpu_manager.go:414] "containerMap: removing container" podUID="c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" containerName="payload-pre-processing"
Oct 31 07:44:47.961596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961514    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" containerName="payload-pre-processing"
Oct 31 07:44:47.961596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961552    5829 cpu_manager.go:414] "containerMap: removing container" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main"
Oct 31 07:44:47.961596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961559    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main"
Oct 31 07:44:47.961795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961717    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b43209-4f04-4826-9f20-a86825f26d63" containerName="revoke-keys"
Oct 31 07:44:47.961795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961734    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f033de-bdf1-430f-94cc-d1fd0b452c9d" containerName="revoke-keys"
Oct 31 07:44:47.961795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961754    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0375cdc-9b5d-468c-8e5f-f989dbc5b5a8" containerName="payload-pre-processing"
Oct 31 07:44:47.961795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.961769    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="a158b083-77df-454a-af0d-e919c4b320c0" containerName="main"
Oct 31 07:44:47.966838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.966813    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:47.986432 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.986399    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"]
Oct 31 07:44:47.990789 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.990749    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:44:47.990926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:47.990822    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:44:48.009560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.009525    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:44:48.009560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.009544    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:44:48.060740 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.060706    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a1f555-afc5-4705-a149-baef955742e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4919ba-xjbhz\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.060926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.060839    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a1f555-afc5-4705-a149-baef955742e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4919ba-xjbhz\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.060994 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.060978    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfkm\" (UniqueName: \"kubernetes.io/projected/72a1f555-afc5-4705-a149-baef955742e6-kube-api-access-lwfkm\") pod \"maas-api-revoke-sub-0b4919ba-xjbhz\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.162517 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.162474    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a1f555-afc5-4705-a149-baef955742e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4919ba-xjbhz\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.162704 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.162628    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfkm\" (UniqueName: \"kubernetes.io/projected/72a1f555-afc5-4705-a149-baef955742e6-kube-api-access-lwfkm\") pod \"maas-api-revoke-sub-0b4919ba-xjbhz\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.162891 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.162860    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a1f555-afc5-4705-a149-baef955742e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4919ba-xjbhz\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.163471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.163447    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a1f555-afc5-4705-a149-baef955742e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b4919ba-xjbhz\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.164999 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.164979    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a1f555-afc5-4705-a149-baef955742e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b4919ba-xjbhz\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.182322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.182274    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfkm\" (UniqueName: \"kubernetes.io/projected/72a1f555-afc5-4705-a149-baef955742e6-kube-api-access-lwfkm\") pod \"maas-api-revoke-sub-0b4919ba-xjbhz\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.287232 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.287146    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:48.601236 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:48.601201    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"]
Oct 31 07:44:48.605438 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:48.605402    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a1f555_afc5_4705_a149_baef955742e6.slice/crio-564d216bf59af38e2c83a90310d6dc5f75bd73f88db98e22087ee8461184be3c WatchSource:0}: Error finding container 564d216bf59af38e2c83a90310d6dc5f75bd73f88db98e22087ee8461184be3c: Status 404 returned error can't find the container with id 564d216bf59af38e2c83a90310d6dc5f75bd73f88db98e22087ee8461184be3c
Oct 31 07:44:49.382007 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:49.381920    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz" event={"ID":"72a1f555-afc5-4705-a149-baef955742e6","Type":"ContainerStarted","Data":"3892c17ccc70947282f99a790df8eb7ac7c2d037dc8af36483abd0a328bf86a6"}
Oct 31 07:44:49.382007 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:49.381977    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz" event={"ID":"72a1f555-afc5-4705-a149-baef955742e6","Type":"ContainerStarted","Data":"564d216bf59af38e2c83a90310d6dc5f75bd73f88db98e22087ee8461184be3c"}
Oct 31 07:44:49.425292 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:49.425228    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz" podStartSLOduration=2.425209347 podStartE2EDuration="2.425209347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:44:49.42041123 +0000 UTC m=+953.559571818" watchObservedRunningTime="2026-10-31 07:44:49.425209347 +0000 UTC m=+953.564369936"
Oct 31 07:44:50.250704 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.247979    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:44:50.331955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.331930    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:44:50.336708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.336680    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:50.408161 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.408124    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-token\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-token\") pod \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") "
Oct 31 07:44:50.409300 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.409172    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a49d057-fed8-4675-8272-8da07af38b1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49d057-fed8-4675-8272-8da07af38b1d-istio-podinfo\") pod \"1a49d057-fed8-4675-8272-8da07af38b1d\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") "
Oct 31 07:44:50.409683 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.409664    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-kube-api-access-5b4pl\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-kube-api-access-5b4pl\") pod \"a8c50054-e3cc-4286-b59f-0a012f93fa92\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") "
Oct 31 07:44:50.409850 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.409832    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-certs\") pod \"a8c50054-e3cc-4286-b59f-0a012f93fa92\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") "
Oct 31 07:44:50.409973 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.409959    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-data\") pod \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") "
Oct 31 07:44:50.410104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.410090    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-podinfo\") pod \"a8c50054-e3cc-4286-b59f-0a012f93fa92\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") "
Oct 31 07:44:50.410220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.410205    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-envoy\") pod \"a8c50054-e3cc-4286-b59f-0a012f93fa92\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") "
Oct 31 07:44:50.410387 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.410370    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a49d057-fed8-4675-8272-8da07af38b1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49d057-fed8-4675-8272-8da07af38b1d-istiod-ca-cert\") pod \"1a49d057-fed8-4675-8272-8da07af38b1d\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") "
Oct 31 07:44:50.410548 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.410531    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c50054-e3cc-4286-b59f-0a012f93fa92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c50054-e3cc-4286-b59f-0a012f93fa92-istiod-ca-cert\") pod \"a8c50054-e3cc-4286-b59f-0a012f93fa92\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") "
Oct 31 07:44:50.410737 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.410720    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-credential-socket\") pod \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") "
Oct 31 07:44:50.410879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.410864    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-socket\") pod \"a8c50054-e3cc-4286-b59f-0a012f93fa92\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") "
Oct 31 07:44:50.411001 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.410987    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-certs\") pod \"1a49d057-fed8-4675-8272-8da07af38b1d\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") "
Oct 31 07:44:50.411164 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.411147    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-socket\") pod \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") "
Oct 31 07:44:50.411318 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.411302    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-envoy\") pod \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") "
Oct 31 07:44:50.411444 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.411430    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-certs\") pod \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") "
Oct 31 07:44:50.411560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.411546    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istiod-ca-cert\") pod \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") "
Oct 31 07:44:50.411721 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.411708    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-data\") pod \"1a49d057-fed8-4675-8272-8da07af38b1d\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") "
Oct 31 07:44:50.411852 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.411839    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-envoy\") pod \"1a49d057-fed8-4675-8272-8da07af38b1d\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") "
Oct 31 07:44:50.411949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.411936    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-credential-socket\") pod \"a8c50054-e3cc-4286-b59f-0a012f93fa92\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") "
Oct 31 07:44:50.412071 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.412055    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-kube-api-access-7vmn2\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-kube-api-access-7vmn2\") pod \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") "
Oct 31 07:44:50.412217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.412190    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-istio-token\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-istio-token\") pod \"1a49d057-fed8-4675-8272-8da07af38b1d\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") "
Oct 31 07:44:50.412328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.412295    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a49d057-fed8-4675-8272-8da07af38b1d-istiod-ca-cert" pod "1a49d057-fed8-4675-8272-8da07af38b1d" (UID: "1a49d057-fed8-4675-8272-8da07af38b1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:50.412419 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.412402    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-kube-api-access-95wbv\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-kube-api-access-95wbv\") pod \"1a49d057-fed8-4675-8272-8da07af38b1d\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") "
Oct 31 07:44:50.412556 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.412542    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-podinfo\") pod \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\" (UID: \"4f3575b5-7dbe-4c72-a538-d50802fc8f5a\") "
Oct 31 07:44:50.412684 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.412669    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-credential-socket\") pod \"1a49d057-fed8-4675-8272-8da07af38b1d\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") "
Oct 31 07:44:50.413841 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.413820    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49d057-fed8-4675-8272-8da07af38b1d-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.414168 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.414137    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-credential-socket" pod "1a49d057-fed8-4675-8272-8da07af38b1d" (UID: "1a49d057-fed8-4675-8272-8da07af38b1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.414732 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.414627    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c50054-e3cc-4286-b59f-0a012f93fa92-istiod-ca-cert" pod "a8c50054-e3cc-4286-b59f-0a012f93fa92" (UID: "a8c50054-e3cc-4286-b59f-0a012f93fa92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:50.418824 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.418796    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-data" pod "1a49d057-fed8-4675-8272-8da07af38b1d" (UID: "1a49d057-fed8-4675-8272-8da07af38b1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.420026 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.419691    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-certs" pod "1a49d057-fed8-4675-8272-8da07af38b1d" (UID: "1a49d057-fed8-4675-8272-8da07af38b1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.420488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.420461    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-credential-socket" pod "4f3575b5-7dbe-4c72-a538-d50802fc8f5a" (UID: "4f3575b5-7dbe-4c72-a538-d50802fc8f5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.420995 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.420973    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-socket" pod "a8c50054-e3cc-4286-b59f-0a012f93fa92" (UID: "a8c50054-e3cc-4286-b59f-0a012f93fa92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.434902 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.422648    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-credential-socket" pod "a8c50054-e3cc-4286-b59f-0a012f93fa92" (UID: "a8c50054-e3cc-4286-b59f-0a012f93fa92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.434902 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.422882    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-socket" pod "4f3575b5-7dbe-4c72-a538-d50802fc8f5a" (UID: "4f3575b5-7dbe-4c72-a538-d50802fc8f5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.435110 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.432864    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-certs" pod "a8c50054-e3cc-4286-b59f-0a012f93fa92" (UID: "a8c50054-e3cc-4286-b59f-0a012f93fa92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.435110 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.433045    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-data" pod "4f3575b5-7dbe-4c72-a538-d50802fc8f5a" (UID: "4f3575b5-7dbe-4c72-a538-d50802fc8f5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.437338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.437294    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-envoy" pod "4f3575b5-7dbe-4c72-a538-d50802fc8f5a" (UID: "4f3575b5-7dbe-4c72-a538-d50802fc8f5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.437709 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.437686    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-certs" pod "4f3575b5-7dbe-4c72-a538-d50802fc8f5a" (UID: "4f3575b5-7dbe-4c72-a538-d50802fc8f5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.438107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.438089    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istiod-ca-cert" pod "4f3575b5-7dbe-4c72-a538-d50802fc8f5a" (UID: "4f3575b5-7dbe-4c72-a538-d50802fc8f5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:50.439715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.439692    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-envoy" pod "1a49d057-fed8-4675-8272-8da07af38b1d" (UID: "1a49d057-fed8-4675-8272-8da07af38b1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.440763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.440719    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-kube-api-access-95wbv" pod "1a49d057-fed8-4675-8272-8da07af38b1d" (UID: "1a49d057-fed8-4675-8272-8da07af38b1d"). InnerVolumeSpecName "kube-api-access-95wbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:50.442860 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.442834    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-podinfo" pod "4f3575b5-7dbe-4c72-a538-d50802fc8f5a" (UID: "4f3575b5-7dbe-4c72-a538-d50802fc8f5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:44:50.452664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.452541    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-podinfo" pod "a8c50054-e3cc-4286-b59f-0a012f93fa92" (UID: "a8c50054-e3cc-4286-b59f-0a012f93fa92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:44:50.452812 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.452746    5829 generic.go:356] "Generic (PLEG): container finished" podID="72a1f555-afc5-4705-a149-baef955742e6" containerID="3892c17ccc70947282f99a790df8eb7ac7c2d037dc8af36483abd0a328bf86a6" exitCode=0
Oct 31 07:44:50.452942 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.452919    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz" event={"ID":"72a1f555-afc5-4705-a149-baef955742e6","Type":"ContainerDied","Data":"3892c17ccc70947282f99a790df8eb7ac7c2d037dc8af36483abd0a328bf86a6"}
Oct 31 07:44:50.453843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.453817    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-envoy" pod "a8c50054-e3cc-4286-b59f-0a012f93fa92" (UID: "a8c50054-e3cc-4286-b59f-0a012f93fa92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.454426 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.454376    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a49d057-fed8-4675-8272-8da07af38b1d-istio-podinfo" pod "1a49d057-fed8-4675-8272-8da07af38b1d" (UID: "1a49d057-fed8-4675-8272-8da07af38b1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:44:50.454546 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.454444    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-token" pod "4f3575b5-7dbe-4c72-a538-d50802fc8f5a" (UID: "4f3575b5-7dbe-4c72-a538-d50802fc8f5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:50.454546 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.454488    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-istio-token" pod "1a49d057-fed8-4675-8272-8da07af38b1d" (UID: "1a49d057-fed8-4675-8272-8da07af38b1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:50.456019 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.455993    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-kube-api-access-5b4pl" pod "a8c50054-e3cc-4286-b59f-0a012f93fa92" (UID: "a8c50054-e3cc-4286-b59f-0a012f93fa92"). InnerVolumeSpecName "kube-api-access-5b4pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:50.456212 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.456193    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-kube-api-access-7vmn2" pod "4f3575b5-7dbe-4c72-a538-d50802fc8f5a" (UID: "4f3575b5-7dbe-4c72-a538-d50802fc8f5a"). InnerVolumeSpecName "kube-api-access-7vmn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:50.456353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.456334    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"
Oct 31 07:44:50.456441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.456212    5829 generic.go:356] "Generic (PLEG): container finished" podID="1a49d057-fed8-4675-8272-8da07af38b1d" containerID="949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f" exitCode=0
Oct 31 07:44:50.456585 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.456237    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" event={"ID":"1a49d057-fed8-4675-8272-8da07af38b1d","Type":"ContainerDied","Data":"949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f"}
Oct 31 07:44:50.456659 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.456623    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc" event={"ID":"1a49d057-fed8-4675-8272-8da07af38b1d","Type":"ContainerDied","Data":"09b03ebe71775dbb63384bb34233566ea44e0e6332bed37cb7fb2a10adf7adf7"}
Oct 31 07:44:50.456721 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.456665    5829 scope.go:122] "RemoveContainer" containerID="949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f"
Oct 31 07:44:50.459004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.458982    5829 generic.go:356] "Generic (PLEG): container finished" podID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerID="8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62" exitCode=0
Oct 31 07:44:50.459148 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.459127    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" event={"ID":"4f3575b5-7dbe-4c72-a538-d50802fc8f5a","Type":"ContainerDied","Data":"8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62"}
Oct 31 07:44:50.459612 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.459592    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8" event={"ID":"4f3575b5-7dbe-4c72-a538-d50802fc8f5a","Type":"ContainerDied","Data":"15b1a6c7cd971f5c7e6b222d57206653121cdd8c84828d39b7b82e0c40473997"}
Oct 31 07:44:50.459820 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.459798    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"
Oct 31 07:44:50.460985 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.460966    5829 generic.go:356] "Generic (PLEG): container finished" podID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerID="2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01" exitCode=0
Oct 31 07:44:50.461341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.461066    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" event={"ID":"a8c50054-e3cc-4286-b59f-0a012f93fa92","Type":"ContainerDied","Data":"2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01"}
Oct 31 07:44:50.461341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.461104    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc" event={"ID":"a8c50054-e3cc-4286-b59f-0a012f93fa92","Type":"ContainerDied","Data":"f8281f3a07b5d946903b93ad18d2bd306d16488858f6202994dd8947544b1d2e"}
Oct 31 07:44:50.461341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.461166    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"
Oct 31 07:44:50.469930 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.469905    5829 scope.go:122] "RemoveContainer" containerID="949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f"
Oct 31 07:44:50.470449 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:50.470401    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f\": container with ID starting with 949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f not found: ID does not exist" containerID="949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f"
Oct 31 07:44:50.470566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.470442    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f"} err="failed to get container status \"949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f\": rpc error: code = NotFound desc = could not find container \"949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f\": container with ID starting with 949bdb4b4168f73dd27ea47805bbfcab78d182f2bdbb43f5b8ab0e91a617012f not found: ID does not exist"
Oct 31 07:44:50.470566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.470468    5829 scope.go:122] "RemoveContainer" containerID="8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62"
Oct 31 07:44:50.482084 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.482059    5829 scope.go:122] "RemoveContainer" containerID="8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62"
Oct 31 07:44:50.482471 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:50.482447    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62\": container with ID starting with 8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62 not found: ID does not exist" containerID="8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62"
Oct 31 07:44:50.482565 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.482480    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62"} err="failed to get container status \"8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62\": rpc error: code = NotFound desc = could not find container \"8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62\": container with ID starting with 8e37a9dd8f081069937111328f9b05072f132c962714a115d80de1a97a944e62 not found: ID does not exist"
Oct 31 07:44:50.482565 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.482510    5829 scope.go:122] "RemoveContainer" containerID="2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01"
Oct 31 07:44:50.494159 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.494126    5829 scope.go:122] "RemoveContainer" containerID="2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01"
Oct 31 07:44:50.494514 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:50.494484    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01\": container with ID starting with 2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01 not found: ID does not exist" containerID="2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01"
Oct 31 07:44:50.494618 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.494522    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01"} err="failed to get container status \"2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01\": rpc error: code = NotFound desc = could not find container \"2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01\": container with ID starting with 2ce78acf502ccdb176ce9d7e73c35c0f05e87eac424fd606b6e48a6cabab6f01 not found: ID does not exist"
Oct 31 07:44:50.514857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.514823    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-token\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-token\") pod \"a8c50054-e3cc-4286-b59f-0a012f93fa92\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") "
Oct 31 07:44:50.514986 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.514874    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-socket\") pod \"1a49d057-fed8-4675-8272-8da07af38b1d\" (UID: \"1a49d057-fed8-4675-8272-8da07af38b1d\") "
Oct 31 07:44:50.514986 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.514925    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-data\") pod \"a8c50054-e3cc-4286-b59f-0a012f93fa92\" (UID: \"a8c50054-e3cc-4286-b59f-0a012f93fa92\") "
Oct 31 07:44:50.515207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515181    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-socket" pod "1a49d057-fed8-4675-8272-8da07af38b1d" (UID: "1a49d057-fed8-4675-8272-8da07af38b1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.515207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515196    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-data" pod "a8c50054-e3cc-4286-b59f-0a012f93fa92" (UID: "a8c50054-e3cc-4286-b59f-0a012f93fa92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:44:50.515606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515586    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515608    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515618    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515626    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49d057-fed8-4675-8272-8da07af38b1d-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515636    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4pl\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-kube-api-access-5b4pl\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515646    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515654    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515663    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515671    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515680    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c50054-e3cc-4286-b59f-0a012f93fa92-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515687    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515696    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515704    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515713    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515720    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515728    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515735    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515743    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515749    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515758    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c50054-e3cc-4286-b59f-0a012f93fa92-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515766    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vmn2\" (UniqueName: \"kubernetes.io/projected/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-kube-api-access-7vmn2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515774    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515782    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95wbv\" (UniqueName: \"kubernetes.io/projected/1a49d057-fed8-4675-8272-8da07af38b1d-kube-api-access-95wbv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515791    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3575b5-7dbe-4c72-a538-d50802fc8f5a-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.515837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.515798    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49d057-fed8-4675-8272-8da07af38b1d-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.517089 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.517066    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-token" pod "a8c50054-e3cc-4286-b59f-0a012f93fa92" (UID: "a8c50054-e3cc-4286-b59f-0a012f93fa92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:50.552677 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.552623    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"]
Oct 31 07:44:50.558185 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.558153    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-jxhp8"]
Oct 31 07:44:50.617581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.617548    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c50054-e3cc-4286-b59f-0a012f93fa92-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:50.807349 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.807312    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"]
Oct 31 07:44:50.812352 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.812321    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-dpbwc"]
Oct 31 07:44:50.845966 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.845929    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"]
Oct 31 07:44:50.851132 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:50.851100    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-648fbe-openshift-default-889c4889f-x8bsc"]
Oct 31 07:44:51.660805 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.660779    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:51.734180 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.734078    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a1f555-afc5-4705-a149-baef955742e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a1f555-afc5-4705-a149-baef955742e6-maas-api-xks-ca\") pod \"72a1f555-afc5-4705-a149-baef955742e6\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") "
Oct 31 07:44:51.734180 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.734150    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a1f555-afc5-4705-a149-baef955742e6-kube-api-access-lwfkm\" (UniqueName: \"kubernetes.io/projected/72a1f555-afc5-4705-a149-baef955742e6-kube-api-access-lwfkm\") pod \"72a1f555-afc5-4705-a149-baef955742e6\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") "
Oct 31 07:44:51.735114 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.735086    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a1f555-afc5-4705-a149-baef955742e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a1f555-afc5-4705-a149-baef955742e6-maas-api-openshift-ca\") pod \"72a1f555-afc5-4705-a149-baef955742e6\" (UID: \"72a1f555-afc5-4705-a149-baef955742e6\") "
Oct 31 07:44:51.735558 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.735530    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a1f555-afc5-4705-a149-baef955742e6-maas-api-openshift-ca" pod "72a1f555-afc5-4705-a149-baef955742e6" (UID: "72a1f555-afc5-4705-a149-baef955742e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:51.735914 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.735891    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72a1f555-afc5-4705-a149-baef955742e6-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:51.736722 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.736700    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a1f555-afc5-4705-a149-baef955742e6-maas-api-xks-ca" pod "72a1f555-afc5-4705-a149-baef955742e6" (UID: "72a1f555-afc5-4705-a149-baef955742e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:44:51.737093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.737068    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a1f555-afc5-4705-a149-baef955742e6-kube-api-access-lwfkm" pod "72a1f555-afc5-4705-a149-baef955742e6" (UID: "72a1f555-afc5-4705-a149-baef955742e6"). InnerVolumeSpecName "kube-api-access-lwfkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:51.837906 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.837871    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72a1f555-afc5-4705-a149-baef955742e6-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:51.837906 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.837901    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwfkm\" (UniqueName: \"kubernetes.io/projected/72a1f555-afc5-4705-a149-baef955742e6-kube-api-access-lwfkm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:51.976536 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.976500    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" path="/var/lib/kubelet/pods/1a49d057-fed8-4675-8272-8da07af38b1d/volumes"
Oct 31 07:44:51.977061 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.977046    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" path="/var/lib/kubelet/pods/4f3575b5-7dbe-4c72-a538-d50802fc8f5a/volumes"
Oct 31 07:44:51.977836 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:51.977820    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" path="/var/lib/kubelet/pods/a8c50054-e3cc-4286-b59f-0a012f93fa92/volumes"
Oct 31 07:44:52.491398 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:52.490472    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz" event={"ID":"72a1f555-afc5-4705-a149-baef955742e6","Type":"ContainerDied","Data":"564d216bf59af38e2c83a90310d6dc5f75bd73f88db98e22087ee8461184be3c"}
Oct 31 07:44:52.491398 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:52.490542    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="564d216bf59af38e2c83a90310d6dc5f75bd73f88db98e22087ee8461184be3c"
Oct 31 07:44:52.491398 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:52.490612    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"
Oct 31 07:44:52.588143 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:52.588099    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"]
Oct 31 07:44:52.643223 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:52.643177    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"]
Oct 31 07:44:54.125305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.124374    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/health\": context deadline exceeded"
Oct 31 07:44:54.214307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.211380    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 07:44:54.214307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.213155    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 07:44:54.281342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.280383    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:54.281342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.281196    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 07:44:54.315308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.313364    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:54.539304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.538384    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:54.539304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.538889    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 07:44:54.913306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.902123    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2" podUID="53782e9d-d7b0-4a17-b756-f14960c29c2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:44:54.913306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.904376    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v" podUID="6e369c6a-79e7-4aed-a1c3-a756e512372c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:44:54.944307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.937010    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:54.991304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:54.990626    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:55.214404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.213382    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:55.214404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.213807    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 07:44:55.214970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.214533    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:44:55.214970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.214597    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:44:55.290901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.290500    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:55.298529 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.291588    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:44:55.312590 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.312547    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:55.313355 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.313254    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:44:55.317971 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.317895    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 31 07:44:55.325383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.325354    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:44:55.585501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.556409    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/ready\": context deadline exceeded"
Oct 31 07:44:55.585501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.556839    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Oct 31 07:44:55.585501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.557487    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:44:55.585501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.557549    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:44:55.990825 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.944599    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:55.990825 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.946158    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:44:56.044464 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.041385    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 31 07:44:56.150697 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:55.710395    5829 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:44:56.233306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.233204    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:44:56.275929 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.239378    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 07:44:56.277968 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.277830    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 07:44:56.296216 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.295977    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 07:44:56.326556 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.312373    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:44:56.326556 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.312822    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:44:56.345894 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.344971    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:44:56.395311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.395252    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:44:56.508742 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.508093    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:44:56.512415 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.512391    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:44:56.516193 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.516171    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:44:56.516460 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.516447    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:44:56.516629 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:56.516612    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:44:57.424991 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.424960    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"]
Oct 31 07:44:57.426077 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426052    5829 cpu_manager.go:414] "containerMap: removing container" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy"
Oct 31 07:44:57.426171 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426081    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy"
Oct 31 07:44:57.426171 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426152    5829 cpu_manager.go:414] "containerMap: removing container" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerName="istio-proxy"
Oct 31 07:44:57.426171 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426161    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerName="istio-proxy"
Oct 31 07:44:57.426362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426188    5829 cpu_manager.go:414] "containerMap: removing container" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy"
Oct 31 07:44:57.426362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426196    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy"
Oct 31 07:44:57.426362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426238    5829 cpu_manager.go:414] "containerMap: removing container" podUID="72a1f555-afc5-4705-a149-baef955742e6" containerName="revoke-keys"
Oct 31 07:44:57.426362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426261    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a1f555-afc5-4705-a149-baef955742e6" containerName="revoke-keys"
Oct 31 07:44:57.426572 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426445    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a1f555-afc5-4705-a149-baef955742e6" containerName="revoke-keys"
Oct 31 07:44:57.426572 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426462    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a49d057-fed8-4675-8272-8da07af38b1d" containerName="istio-proxy"
Oct 31 07:44:57.426572 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426485    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3575b5-7dbe-4c72-a538-d50802fc8f5a" containerName="istio-proxy"
Oct 31 07:44:57.426572 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.426498    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c50054-e3cc-4286-b59f-0a012f93fa92" containerName="istio-proxy"
Oct 31 07:44:57.433228 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.433200    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.447720 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.447684    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"]
Oct 31 07:44:57.452093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.451734    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:44:57.452093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.451748    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:44:57.452093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.451778    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:44:57.452093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.451838    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:44:57.504632 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.504600    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.504886 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.504864    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.505024 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.504996    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbcf4\" (UniqueName: \"kubernetes.io/projected/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-kube-api-access-kbcf4\") pod \"maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.589107 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.589057    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726" podUID="8f6c6f59-6834-4d62-9e21-1f263fff5b7b" containerName="payload-processing" containerID="cri-o://f78ba7606153235b5125ad56dea27f82241bc7bb195332807e8048cf797cc5b1" gracePeriod=25
Oct 31 07:44:57.606363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.606331    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.606609 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.606574    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.606673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.606652    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbcf4\" (UniqueName: \"kubernetes.io/projected/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-kube-api-access-kbcf4\") pod \"maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.607246 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.607219    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.608958 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.608933    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.626672 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.626600    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbcf4\" (UniqueName: \"kubernetes.io/projected/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-kube-api-access-kbcf4\") pod \"maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:57.644394 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.644341    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f" podUID="91c0428e-b966-4e6e-8425-15bc33a040b6" containerName="payload-pre-processing" containerID="cri-o://179a509c109305b296d36c07e45dd5fe48277e0ff7775fba63ff7d076bad7f47" gracePeriod=25
Oct 31 07:44:57.756234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:57.756195    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:44:58.077954 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.077920    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"]
Oct 31 07:44:58.081039 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:58.080950    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5bd411_2ab6_4a3b_bdfa_add465dd3f16.slice/crio-a828a10b66be9229c133fe098d5325205c7af6bcbf64fb017912acd81b1be10a WatchSource:0}: Error finding container a828a10b66be9229c133fe098d5325205c7af6bcbf64fb017912acd81b1be10a: Status 404 returned error can't find the container with id a828a10b66be9229c133fe098d5325205c7af6bcbf64fb017912acd81b1be10a
Oct 31 07:44:58.325995 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.325963    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:44:58.338475 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.338447    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:44:58.392967 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.392927    5829 generic.go:356] "Generic (PLEG): container finished" podID="53782e9d-d7b0-4a17-b756-f14960c29c2c" containerID="7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76" exitCode=0
Oct 31 07:44:58.392967 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.392965    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"
Oct 31 07:44:58.393217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.393000    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2" event={"ID":"53782e9d-d7b0-4a17-b756-f14960c29c2c","Type":"ContainerDied","Data":"7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76"}
Oct 31 07:44:58.393217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.393090    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2" event={"ID":"53782e9d-d7b0-4a17-b756-f14960c29c2c","Type":"ContainerDied","Data":"9112a856d9e06b69379c7b2d05ad81ddb4579a7cff25af20c8b5a6de5092dbf9"}
Oct 31 07:44:58.393217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.393119    5829 scope.go:122] "RemoveContainer" containerID="7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76"
Oct 31 07:44:58.395047 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.395021    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk" event={"ID":"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16","Type":"ContainerStarted","Data":"a828a10b66be9229c133fe098d5325205c7af6bcbf64fb017912acd81b1be10a"}
Oct 31 07:44:58.396977 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.396951    5829 generic.go:356] "Generic (PLEG): container finished" podID="6e369c6a-79e7-4aed-a1c3-a756e512372c" containerID="4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac" exitCode=0
Oct 31 07:44:58.397102 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.396995    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v" event={"ID":"6e369c6a-79e7-4aed-a1c3-a756e512372c","Type":"ContainerDied","Data":"4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac"}
Oct 31 07:44:58.397102 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.397031    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v" event={"ID":"6e369c6a-79e7-4aed-a1c3-a756e512372c","Type":"ContainerDied","Data":"1329df141b09de736af671f67f7786633e30f50bd5dc598dd336bc4bc867a9a0"}
Oct 31 07:44:58.397102 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.397039    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"
Oct 31 07:44:58.405903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.405874    5829 scope.go:122] "RemoveContainer" containerID="7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76"
Oct 31 07:44:58.406324 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:58.406303    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76\": container with ID starting with 7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76 not found: ID does not exist" containerID="7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76"
Oct 31 07:44:58.406416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.406331    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76"} err="failed to get container status \"7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76\": rpc error: code = NotFound desc = could not find container \"7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76\": container with ID starting with 7d10d1e17fc0da0c1c5d8da5d1aee807185e973acc92258ab07306ffe8f88b76 not found: ID does not exist"
Oct 31 07:44:58.406416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.406358    5829 scope.go:122] "RemoveContainer" containerID="4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac"
Oct 31 07:44:58.419616 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.419565    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53782e9d-d7b0-4a17-b756-f14960c29c2c-kube-api-access-z8scc\" (UniqueName: \"kubernetes.io/projected/53782e9d-d7b0-4a17-b756-f14960c29c2c-kube-api-access-z8scc\") pod \"53782e9d-d7b0-4a17-b756-f14960c29c2c\" (UID: \"53782e9d-d7b0-4a17-b756-f14960c29c2c\") "
Oct 31 07:44:58.420091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.420057    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53782e9d-d7b0-4a17-b756-f14960c29c2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53782e9d-d7b0-4a17-b756-f14960c29c2c-plugins-config-volume\") pod \"53782e9d-d7b0-4a17-b756-f14960c29c2c\" (UID: \"53782e9d-d7b0-4a17-b756-f14960c29c2c\") "
Oct 31 07:44:58.420217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.420168    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e369c6a-79e7-4aed-a1c3-a756e512372c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e369c6a-79e7-4aed-a1c3-a756e512372c-plugins-config-volume\") pod \"6e369c6a-79e7-4aed-a1c3-a756e512372c\" (UID: \"6e369c6a-79e7-4aed-a1c3-a756e512372c\") "
Oct 31 07:44:58.420340 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.420328    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e369c6a-79e7-4aed-a1c3-a756e512372c-kube-api-access-jmrmf\" (UniqueName: \"kubernetes.io/projected/6e369c6a-79e7-4aed-a1c3-a756e512372c-kube-api-access-jmrmf\") pod \"6e369c6a-79e7-4aed-a1c3-a756e512372c\" (UID: \"6e369c6a-79e7-4aed-a1c3-a756e512372c\") "
Oct 31 07:44:58.421854 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.421821    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e369c6a-79e7-4aed-a1c3-a756e512372c-plugins-config-volume" pod "6e369c6a-79e7-4aed-a1c3-a756e512372c" (UID: "6e369c6a-79e7-4aed-a1c3-a756e512372c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:58.423731 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.423696    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53782e9d-d7b0-4a17-b756-f14960c29c2c-plugins-config-volume" pod "53782e9d-d7b0-4a17-b756-f14960c29c2c" (UID: "53782e9d-d7b0-4a17-b756-f14960c29c2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:44:58.427620 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.427592    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e369c6a-79e7-4aed-a1c3-a756e512372c-kube-api-access-jmrmf" pod "6e369c6a-79e7-4aed-a1c3-a756e512372c" (UID: "6e369c6a-79e7-4aed-a1c3-a756e512372c"). InnerVolumeSpecName "kube-api-access-jmrmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:58.428425 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.428401    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53782e9d-d7b0-4a17-b756-f14960c29c2c-kube-api-access-z8scc" pod "53782e9d-d7b0-4a17-b756-f14960c29c2c" (UID: "53782e9d-d7b0-4a17-b756-f14960c29c2c"). InnerVolumeSpecName "kube-api-access-z8scc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:44:58.435499 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.435476    5829 scope.go:122] "RemoveContainer" containerID="4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac"
Oct 31 07:44:58.435906 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:44:58.435878    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac\": container with ID starting with 4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac not found: ID does not exist" containerID="4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac"
Oct 31 07:44:58.435995 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.435923    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac"} err="failed to get container status \"4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac\": rpc error: code = NotFound desc = could not find container \"4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac\": container with ID starting with 4afe6fa775657d711bff374f7edb88b66bac8ae2a694f44e3e98aecb0b373dac not found: ID does not exist"
Oct 31 07:44:58.523003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.522967    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8scc\" (UniqueName: \"kubernetes.io/projected/53782e9d-d7b0-4a17-b756-f14960c29c2c-kube-api-access-z8scc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:58.523003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.522997    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53782e9d-d7b0-4a17-b756-f14960c29c2c-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:58.523003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.523010    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e369c6a-79e7-4aed-a1c3-a756e512372c-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:58.523247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.523020    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmrmf\" (UniqueName: \"kubernetes.io/projected/6e369c6a-79e7-4aed-a1c3-a756e512372c-kube-api-access-jmrmf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:44:58.744271 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.744234    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"]
Oct 31 07:44:58.749716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.749684    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-027d2a-59c86d9fc4-qqmm2"]
Oct 31 07:44:58.783781 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.783750    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"]
Oct 31 07:44:58.788699 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:58.788668    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-027d2a-76bbdf5c5-r582v"]
Oct 31 07:44:59.332752 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.332715    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"]
Oct 31 07:44:59.333452 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.333437    5829 cpu_manager.go:414] "containerMap: removing container" podUID="53782e9d-d7b0-4a17-b756-f14960c29c2c" containerName="payload-processing"
Oct 31 07:44:59.333526 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.333453    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53782e9d-d7b0-4a17-b756-f14960c29c2c" containerName="payload-processing"
Oct 31 07:44:59.333526 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.333469    5829 cpu_manager.go:414] "containerMap: removing container" podUID="6e369c6a-79e7-4aed-a1c3-a756e512372c" containerName="payload-pre-processing"
Oct 31 07:44:59.333526 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.333474    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e369c6a-79e7-4aed-a1c3-a756e512372c" containerName="payload-pre-processing"
Oct 31 07:44:59.333631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.333616    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="53782e9d-d7b0-4a17-b756-f14960c29c2c" containerName="payload-processing"
Oct 31 07:44:59.333631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.333626    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e369c6a-79e7-4aed-a1c3-a756e512372c" containerName="payload-pre-processing"
Oct 31 07:44:59.337563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.337543    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.373186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.373154    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"]
Oct 31 07:44:59.411860 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.411768    5829 generic.go:356] "Generic (PLEG): container finished" podID="8a5bd411-2ab6-4a3b-bdfa-add465dd3f16" containerID="8ed661cc63c7c671372f0f819bf7d5c6651856bdcdf40e6ab8c745c31e000b3d" exitCode=0
Oct 31 07:44:59.412021 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.411848    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk" event={"ID":"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16","Type":"ContainerDied","Data":"8ed661cc63c7c671372f0f819bf7d5c6651856bdcdf40e6ab8c745c31e000b3d"}
Oct 31 07:44:59.437745 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.437711    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcvvh\" (UniqueName: \"kubernetes.io/projected/b75a8cf6-1560-4982-9c99-6d59f996512b-kube-api-access-dcvvh\") pod \"maas-api-revoke-sub-980d26c0-7j78w\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.438135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.437781    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-980d26c0-7j78w\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.438135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.437867    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-980d26c0-7j78w\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.539102 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.539065    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcvvh\" (UniqueName: \"kubernetes.io/projected/b75a8cf6-1560-4982-9c99-6d59f996512b-kube-api-access-dcvvh\") pod \"maas-api-revoke-sub-980d26c0-7j78w\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.539327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.539135    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-980d26c0-7j78w\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.539327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.539180    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-980d26c0-7j78w\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.539931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.539907    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-980d26c0-7j78w\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.541584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.541560    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-980d26c0-7j78w\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.558685 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.558655    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcvvh\" (UniqueName: \"kubernetes.io/projected/b75a8cf6-1560-4982-9c99-6d59f996512b-kube-api-access-dcvvh\") pod \"maas-api-revoke-sub-980d26c0-7j78w\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.653126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.653097    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:44:59.959065 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:44:59.959027    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75a8cf6_1560_4982_9c99_6d59f996512b.slice/crio-7bcc70cf6c863bde31c4380612f561096162b36ab191f5c81ff8a0e4d63c7160 WatchSource:0}: Error finding container 7bcc70cf6c863bde31c4380612f561096162b36ab191f5c81ff8a0e4d63c7160: Status 404 returned error can't find the container with id 7bcc70cf6c863bde31c4380612f561096162b36ab191f5c81ff8a0e4d63c7160
Oct 31 07:44:59.960083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.960057    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"]
Oct 31 07:44:59.980293 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.980254    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53782e9d-d7b0-4a17-b756-f14960c29c2c" path="/var/lib/kubelet/pods/53782e9d-d7b0-4a17-b756-f14960c29c2c/volumes"
Oct 31 07:44:59.980806 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:44:59.980789    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e369c6a-79e7-4aed-a1c3-a756e512372c" path="/var/lib/kubelet/pods/6e369c6a-79e7-4aed-a1c3-a756e512372c/volumes"
Oct 31 07:45:00.139801 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.139757    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"]
Oct 31 07:45:00.165471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.165434    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"]
Oct 31 07:45:00.165784 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.165761    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:00.172892 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.172861    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"]
Oct 31 07:45:00.173345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.173258    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:00.181020 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.180994    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"]
Oct 31 07:45:00.181230 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.181102    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:00.184726 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.184694    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"]
Oct 31 07:45:00.192148 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.192121    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"]
Oct 31 07:45:00.255069 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.255036    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv\" (UID: \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:00.255222 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.255104    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpjc\" (UniqueName: \"kubernetes.io/projected/993617de-c2f4-4f68-899b-f4cd1b192429-kube-api-access-wcpjc\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd\" (UID: \"993617de-c2f4-4f68-899b-f4cd1b192429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:00.255334 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.255262    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965a35f-4b30-4e02-b22f-62634ae83b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl\" (UID: \"8965a35f-4b30-4e02-b22f-62634ae83b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:00.255601 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.255577    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmzf\" (UniqueName: \"kubernetes.io/projected/8965a35f-4b30-4e02-b22f-62634ae83b31-kube-api-access-ljmzf\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl\" (UID: \"8965a35f-4b30-4e02-b22f-62634ae83b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:00.255693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.255644    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993617de-c2f4-4f68-899b-f4cd1b192429-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd\" (UID: \"993617de-c2f4-4f68-899b-f4cd1b192429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:00.255693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.255673    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrdqx\" (UniqueName: \"kubernetes.io/projected/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-kube-api-access-nrdqx\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv\" (UID: \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:00.356928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.356887    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993617de-c2f4-4f68-899b-f4cd1b192429-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd\" (UID: \"993617de-c2f4-4f68-899b-f4cd1b192429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:00.357123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.356945    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrdqx\" (UniqueName: \"kubernetes.io/projected/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-kube-api-access-nrdqx\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv\" (UID: \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:00.357123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.357075    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv\" (UID: \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:00.357251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.357155    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpjc\" (UniqueName: \"kubernetes.io/projected/993617de-c2f4-4f68-899b-f4cd1b192429-kube-api-access-wcpjc\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd\" (UID: \"993617de-c2f4-4f68-899b-f4cd1b192429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:00.357251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.357207    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965a35f-4b30-4e02-b22f-62634ae83b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl\" (UID: \"8965a35f-4b30-4e02-b22f-62634ae83b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:00.357519 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.357483    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmzf\" (UniqueName: \"kubernetes.io/projected/8965a35f-4b30-4e02-b22f-62634ae83b31-kube-api-access-ljmzf\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl\" (UID: \"8965a35f-4b30-4e02-b22f-62634ae83b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:00.358174 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.358148    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv\" (UID: \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:00.358424 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.358397    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965a35f-4b30-4e02-b22f-62634ae83b31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl\" (UID: \"8965a35f-4b30-4e02-b22f-62634ae83b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:00.358529 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.358450    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993617de-c2f4-4f68-899b-f4cd1b192429-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd\" (UID: \"993617de-c2f4-4f68-899b-f4cd1b192429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:00.377619 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.377589    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrdqx\" (UniqueName: \"kubernetes.io/projected/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-kube-api-access-nrdqx\") pod \"maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv\" (UID: \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:00.378003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.377978    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmzf\" (UniqueName: \"kubernetes.io/projected/8965a35f-4b30-4e02-b22f-62634ae83b31-kube-api-access-ljmzf\") pod \"maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl\" (UID: \"8965a35f-4b30-4e02-b22f-62634ae83b31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:00.378898 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.378872    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpjc\" (UniqueName: \"kubernetes.io/projected/993617de-c2f4-4f68-899b-f4cd1b192429-kube-api-access-wcpjc\") pod \"maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd\" (UID: \"993617de-c2f4-4f68-899b-f4cd1b192429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:00.423339 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.423265    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w" event={"ID":"b75a8cf6-1560-4982-9c99-6d59f996512b","Type":"ContainerStarted","Data":"7bcc70cf6c863bde31c4380612f561096162b36ab191f5c81ff8a0e4d63c7160"}
Oct 31 07:45:00.497567 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.497522    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:00.506911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.506883    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:00.530787 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.530563    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:00.573936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.573904    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:45:00.662035 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.661998    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-kube-api-access-kbcf4\" (UniqueName: \"kubernetes.io/projected/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-kube-api-access-kbcf4\") pod \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") "
Oct 31 07:45:00.662199 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.662129    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-openshift-ca\") pod \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") "
Oct 31 07:45:00.662542 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.662512    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-xks-ca\") pod \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\" (UID: \"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16\") "
Oct 31 07:45:00.662542 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.662529    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-openshift-ca" pod "8a5bd411-2ab6-4a3b-bdfa-add465dd3f16" (UID: "8a5bd411-2ab6-4a3b-bdfa-add465dd3f16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:00.663139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.663115    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:00.665234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.665196    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-kube-api-access-kbcf4" pod "8a5bd411-2ab6-4a3b-bdfa-add465dd3f16" (UID: "8a5bd411-2ab6-4a3b-bdfa-add465dd3f16"). InnerVolumeSpecName "kube-api-access-kbcf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:00.665350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.665237    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-xks-ca" pod "8a5bd411-2ab6-4a3b-bdfa-add465dd3f16" (UID: "8a5bd411-2ab6-4a3b-bdfa-add465dd3f16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:00.765240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.765147    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:00.765240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.765193    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbcf4\" (UniqueName: \"kubernetes.io/projected/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16-kube-api-access-kbcf4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:00.851394 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.851360    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"]
Oct 31 07:45:00.851575 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:45:00.851441    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993617de_c2f4_4f68_899b_f4cd1b192429.slice/crio-7b7fd5928909dd90d780e37afc8496f10ac295f9e6f833295b082a45e9ec1694 WatchSource:0}: Error finding container 7b7fd5928909dd90d780e37afc8496f10ac295f9e6f833295b082a45e9ec1694: Status 404 returned error can't find the container with id 7b7fd5928909dd90d780e37afc8496f10ac295f9e6f833295b082a45e9ec1694
Oct 31 07:45:00.866581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.866551    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"]
Oct 31 07:45:00.867401 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:45:00.867365    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd35962de_8dfc_477e_aebe_c0fb2ec68e5e.slice/crio-30857cbfec64b08d7b70f34c6b27a61bc946e96414ee173ea391031536e2e187 WatchSource:0}: Error finding container 30857cbfec64b08d7b70f34c6b27a61bc946e96414ee173ea391031536e2e187: Status 404 returned error can't find the container with id 30857cbfec64b08d7b70f34c6b27a61bc946e96414ee173ea391031536e2e187
Oct 31 07:45:00.887080 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:45:00.887044    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8965a35f_4b30_4e02_b22f_62634ae83b31.slice/crio-8b038277282034fdfe24e09e53a78dd25bcd36ce143919e3dab2c17ddceab4bb WatchSource:0}: Error finding container 8b038277282034fdfe24e09e53a78dd25bcd36ce143919e3dab2c17ddceab4bb: Status 404 returned error can't find the container with id 8b038277282034fdfe24e09e53a78dd25bcd36ce143919e3dab2c17ddceab4bb
Oct 31 07:45:00.888954 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.888927    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"]
Oct 31 07:45:00.915752 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.915678    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726" podUID="8f6c6f59-6834-4d62-9e21-1f263fff5b7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:00.915752 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:00.915682    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f" podUID="91c0428e-b966-4e6e-8425-15bc33a040b6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:01.456953 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.456913    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"]
Oct 31 07:45:01.458486 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.458456    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8a5bd411-2ab6-4a3b-bdfa-add465dd3f16" containerName="revoke-keys"
Oct 31 07:45:01.458668 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.458654    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5bd411-2ab6-4a3b-bdfa-add465dd3f16" containerName="revoke-keys"
Oct 31 07:45:01.458977 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.458963    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5bd411-2ab6-4a3b-bdfa-add465dd3f16" containerName="revoke-keys"
Oct 31 07:45:01.463903 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.463868    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" event={"ID":"993617de-c2f4-4f68-899b-f4cd1b192429","Type":"ContainerStarted","Data":"3027478f8cebf23ecb70a55e95d76a783b835b92d7bc3c9a156fd53d1918d634"}
Oct 31 07:45:01.464027 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.463916    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" event={"ID":"993617de-c2f4-4f68-899b-f4cd1b192429","Type":"ContainerStarted","Data":"7b7fd5928909dd90d780e37afc8496f10ac295f9e6f833295b082a45e9ec1694"}
Oct 31 07:45:01.464027 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.464013    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.464511 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.464481    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" event={"ID":"8965a35f-4b30-4e02-b22f-62634ae83b31","Type":"ContainerStarted","Data":"8b038277282034fdfe24e09e53a78dd25bcd36ce143919e3dab2c17ddceab4bb"}
Oct 31 07:45:01.466644 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.466619    5829 generic.go:356] "Generic (PLEG): container finished" podID="b75a8cf6-1560-4982-9c99-6d59f996512b" containerID="d1d61c8970ef405e25ae6d4cff9c0b0d9617e08fa444f492a63c34b4d40bd7f7" exitCode=0
Oct 31 07:45:01.466802 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.466789    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w" event={"ID":"b75a8cf6-1560-4982-9c99-6d59f996512b","Type":"ContainerDied","Data":"d1d61c8970ef405e25ae6d4cff9c0b0d9617e08fa444f492a63c34b4d40bd7f7"}
Oct 31 07:45:01.468773 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.468752    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"
Oct 31 07:45:01.469122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.469039    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk" event={"ID":"8a5bd411-2ab6-4a3b-bdfa-add465dd3f16","Type":"ContainerDied","Data":"a828a10b66be9229c133fe098d5325205c7af6bcbf64fb017912acd81b1be10a"}
Oct 31 07:45:01.469122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.469089    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a828a10b66be9229c133fe098d5325205c7af6bcbf64fb017912acd81b1be10a"
Oct 31 07:45:01.470968 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.470938    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" event={"ID":"d35962de-8dfc-477e-aebe-c0fb2ec68e5e","Type":"ContainerStarted","Data":"9dc55395d1909318a044edebcd5805aef65d0e8768bc960bd948e3a5ad163639"}
Oct 31 07:45:01.471110 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.470990    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" event={"ID":"d35962de-8dfc-477e-aebe-c0fb2ec68e5e","Type":"ContainerStarted","Data":"30857cbfec64b08d7b70f34c6b27a61bc946e96414ee173ea391031536e2e187"}
Oct 31 07:45:01.476308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.476269    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"]
Oct 31 07:45:01.560791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.560728    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" podStartSLOduration=1.560705756 podStartE2EDuration="1.560705756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:01.514934919 +0000 UTC m=+965.654095509" watchObservedRunningTime="2026-10-31 07:45:01.560705756 +0000 UTC m=+965.699866344"
Oct 31 07:45:01.590640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.590602    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mrq\" (UniqueName: \"kubernetes.io/projected/0a6519d7-c1dc-40e4-b83a-a8a07670662c-kube-api-access-96mrq\") pod \"maas-api-revoke-sub-e93e4474-c6l5h\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.591030 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.590997    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e93e4474-c6l5h\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.591223 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.591194    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e93e4474-c6l5h\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.613294 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.603803    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"]
Oct 31 07:45:01.613294 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.604358    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"]
Oct 31 07:45:01.691618 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.686250    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" podStartSLOduration=1.68622901 podStartE2EDuration="1.68622901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:01.636914913 +0000 UTC m=+965.776075523" watchObservedRunningTime="2026-10-31 07:45:01.68622901 +0000 UTC m=+965.825389599"
Oct 31 07:45:01.696735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.696596    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mrq\" (UniqueName: \"kubernetes.io/projected/0a6519d7-c1dc-40e4-b83a-a8a07670662c-kube-api-access-96mrq\") pod \"maas-api-revoke-sub-e93e4474-c6l5h\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.696897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.696745    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e93e4474-c6l5h\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.696897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.696809    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e93e4474-c6l5h\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.739534 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.705856    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e93e4474-c6l5h\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.739534 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.709303    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e93e4474-c6l5h\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.745791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.745466    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" containerID="cri-o://19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24" gracePeriod=45
Oct 31 07:45:01.761177 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.760444    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mrq\" (UniqueName: \"kubernetes.io/projected/0a6519d7-c1dc-40e4-b83a-a8a07670662c-kube-api-access-96mrq\") pod \"maas-api-revoke-sub-e93e4474-c6l5h\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.762844 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.762611    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" containerID="cri-o://4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7" gracePeriod=45
Oct 31 07:45:01.783316 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.783234    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:01.939349 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:01.938103    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": dial tcp 10.134.0.54:8000: connect: connection refused"
Oct 31 07:45:02.131371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.127634    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"]
Oct 31 07:45:02.193098 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.193028    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"]
Oct 31 07:45:02.218358 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.211094    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-027d2a-tmpqp"]
Oct 31 07:45:02.312312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.311860    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:45:02.334778 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.334738    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-home\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-home\") pod \"dc39e212-ced0-4425-a28e-e38ebfc09652\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") "
Oct 31 07:45:02.334945 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.334895    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc39e212-ced0-4425-a28e-e38ebfc09652-kube-api-access-68hm2\" (UniqueName: \"kubernetes.io/projected/dc39e212-ced0-4425-a28e-e38ebfc09652-kube-api-access-68hm2\") pod \"dc39e212-ced0-4425-a28e-e38ebfc09652\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") "
Oct 31 07:45:02.335023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.334951    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-dshm\") pod \"dc39e212-ced0-4425-a28e-e38ebfc09652\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") "
Oct 31 07:45:02.335143 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.335037    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc39e212-ced0-4425-a28e-e38ebfc09652-tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39e212-ced0-4425-a28e-e38ebfc09652-tls-certs\") pod \"dc39e212-ced0-4425-a28e-e38ebfc09652\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") "
Oct 31 07:45:02.335143 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.335116    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-tmp-dir\") pod \"dc39e212-ced0-4425-a28e-e38ebfc09652\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") "
Oct 31 07:45:02.335259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.335156    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-model-cache\") pod \"dc39e212-ced0-4425-a28e-e38ebfc09652\" (UID: \"dc39e212-ced0-4425-a28e-e38ebfc09652\") "
Oct 31 07:45:02.336427 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.335478    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:45:02.336569 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.336433    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-model-cache" pod "dc39e212-ced0-4425-a28e-e38ebfc09652" (UID: "dc39e212-ced0-4425-a28e-e38ebfc09652"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:02.342306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.336691    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-home" pod "dc39e212-ced0-4425-a28e-e38ebfc09652" (UID: "dc39e212-ced0-4425-a28e-e38ebfc09652"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:02.342306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.339730    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-tmp-dir" pod "dc39e212-ced0-4425-a28e-e38ebfc09652" (UID: "dc39e212-ced0-4425-a28e-e38ebfc09652"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:02.417471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.417408    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc39e212-ced0-4425-a28e-e38ebfc09652-kube-api-access-68hm2" pod "dc39e212-ced0-4425-a28e-e38ebfc09652" (UID: "dc39e212-ced0-4425-a28e-e38ebfc09652"). InnerVolumeSpecName "kube-api-access-68hm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:02.417646 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.417522    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-dshm" pod "dc39e212-ced0-4425-a28e-e38ebfc09652" (UID: "dc39e212-ced0-4425-a28e-e38ebfc09652"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:02.417646 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.417599    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc39e212-ced0-4425-a28e-e38ebfc09652-tls-certs" pod "dc39e212-ced0-4425-a28e-e38ebfc09652" (UID: "dc39e212-ced0-4425-a28e-e38ebfc09652"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:02.439597 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.439557    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-tmp-dir\") pod \"70adf136-b28a-439e-b53f-afce743b4a79\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") "
Oct 31 07:45:02.439888 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.439866    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70adf136-b28a-439e-b53f-afce743b4a79-tls-certs\" (UniqueName: \"kubernetes.io/secret/70adf136-b28a-439e-b53f-afce743b4a79-tls-certs\") pod \"70adf136-b28a-439e-b53f-afce743b4a79\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") "
Oct 31 07:45:02.440120 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.440097    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-home\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-home\") pod \"70adf136-b28a-439e-b53f-afce743b4a79\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") "
Oct 31 07:45:02.440370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.440327    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70adf136-b28a-439e-b53f-afce743b4a79-kube-api-access-8fcbt\" (UniqueName: \"kubernetes.io/projected/70adf136-b28a-439e-b53f-afce743b4a79-kube-api-access-8fcbt\") pod \"70adf136-b28a-439e-b53f-afce743b4a79\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") "
Oct 31 07:45:02.440523 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.440503    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-model-cache\") pod \"70adf136-b28a-439e-b53f-afce743b4a79\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") "
Oct 31 07:45:02.441358 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.441332    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.441566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.441549    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68hm2\" (UniqueName: \"kubernetes.io/projected/dc39e212-ced0-4425-a28e-e38ebfc09652-kube-api-access-68hm2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.441700 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.441687    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.441831 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.441816    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc39e212-ced0-4425-a28e-e38ebfc09652-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.441988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.441975    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.442115 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.442104    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc39e212-ced0-4425-a28e-e38ebfc09652-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.442236 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.439888    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-tmp-dir" pod "70adf136-b28a-439e-b53f-afce743b4a79" (UID: "70adf136-b28a-439e-b53f-afce743b4a79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:02.442396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.440358    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-home" pod "70adf136-b28a-439e-b53f-afce743b4a79" (UID: "70adf136-b28a-439e-b53f-afce743b4a79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:02.442485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.441625    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-model-cache" pod "70adf136-b28a-439e-b53f-afce743b4a79" (UID: "70adf136-b28a-439e-b53f-afce743b4a79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:02.442773 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.442741    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70adf136-b28a-439e-b53f-afce743b4a79-tls-certs" pod "70adf136-b28a-439e-b53f-afce743b4a79" (UID: "70adf136-b28a-439e-b53f-afce743b4a79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:02.450581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.450526    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70adf136-b28a-439e-b53f-afce743b4a79-kube-api-access-8fcbt" pod "70adf136-b28a-439e-b53f-afce743b4a79" (UID: "70adf136-b28a-439e-b53f-afce743b4a79"). InnerVolumeSpecName "kube-api-access-8fcbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:02.486890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.486854    5829 generic.go:356] "Generic (PLEG): container finished" podID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerID="9dc55395d1909318a044edebcd5805aef65d0e8768bc960bd948e3a5ad163639" exitCode=60
Oct 31 07:45:02.487093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.486938    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" event={"ID":"d35962de-8dfc-477e-aebe-c0fb2ec68e5e","Type":"ContainerDied","Data":"9dc55395d1909318a044edebcd5805aef65d0e8768bc960bd948e3a5ad163639"}
Oct 31 07:45:02.487336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.487318    5829 scope.go:122] "RemoveContainer" containerID="9dc55395d1909318a044edebcd5805aef65d0e8768bc960bd948e3a5ad163639"
Oct 31 07:45:02.488978 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.488952    5829 generic.go:356] "Generic (PLEG): container finished" podID="993617de-c2f4-4f68-899b-f4cd1b192429" containerID="3027478f8cebf23ecb70a55e95d76a783b835b92d7bc3c9a156fd53d1918d634" exitCode=60
Oct 31 07:45:02.489163 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.488992    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" event={"ID":"993617de-c2f4-4f68-899b-f4cd1b192429","Type":"ContainerDied","Data":"3027478f8cebf23ecb70a55e95d76a783b835b92d7bc3c9a156fd53d1918d634"}
Oct 31 07:45:02.489428 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.489408    5829 scope.go:122] "RemoveContainer" containerID="3027478f8cebf23ecb70a55e95d76a783b835b92d7bc3c9a156fd53d1918d634"
Oct 31 07:45:02.490787 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.490761    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h" event={"ID":"0a6519d7-c1dc-40e4-b83a-a8a07670662c","Type":"ContainerStarted","Data":"f4506d72ffa11c5a3d0057d9aa9b033f74a89e94dc220ad3c1947f8fea3e28d4"}
Oct 31 07:45:02.492837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.492811    5829 generic.go:356] "Generic (PLEG): container finished" podID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerID="d7966bb2aa911cd673fcd91a6ebf5d7f6c153553a275c5bab4808ae1e37e55f5" exitCode=60
Oct 31 07:45:02.492959 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.492933    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" event={"ID":"8965a35f-4b30-4e02-b22f-62634ae83b31","Type":"ContainerDied","Data":"d7966bb2aa911cd673fcd91a6ebf5d7f6c153553a275c5bab4808ae1e37e55f5"}
Oct 31 07:45:02.493205 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.493155    5829 scope.go:122] "RemoveContainer" containerID="d7966bb2aa911cd673fcd91a6ebf5d7f6c153553a275c5bab4808ae1e37e55f5"
Oct 31 07:45:02.494837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.494816    5829 generic.go:356] "Generic (PLEG): container finished" podID="70adf136-b28a-439e-b53f-afce743b4a79" containerID="4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7" exitCode=0
Oct 31 07:45:02.494936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.494879    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"
Oct 31 07:45:02.494988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.494961    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" event={"ID":"70adf136-b28a-439e-b53f-afce743b4a79","Type":"ContainerDied","Data":"4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7"}
Oct 31 07:45:02.495029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.494987    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" event={"ID":"70adf136-b28a-439e-b53f-afce743b4a79","Type":"ContainerDied","Data":"ecdc62cdd0630ec9e464b14cd50902eb6863bba36e573d56be62443f15c35102"}
Oct 31 07:45:02.495029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.495002    5829 scope.go:122] "RemoveContainer" containerID="4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7"
Oct 31 07:45:02.496892 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.496868    5829 generic.go:356] "Generic (PLEG): container finished" podID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerID="19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24" exitCode=0
Oct 31 07:45:02.497076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.496921    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"
Oct 31 07:45:02.497192 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.496921    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" event={"ID":"dc39e212-ced0-4425-a28e-e38ebfc09652","Type":"ContainerDied","Data":"19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24"}
Oct 31 07:45:02.497248 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.497210    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb" event={"ID":"dc39e212-ced0-4425-a28e-e38ebfc09652","Type":"ContainerDied","Data":"11aaf94606c340d7e694342bca6f6205e62a6148794d455f11b229c07d677bdb"}
Oct 31 07:45:02.543309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.543087    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-dshm\") pod \"70adf136-b28a-439e-b53f-afce743b4a79\" (UID: \"70adf136-b28a-439e-b53f-afce743b4a79\") "
Oct 31 07:45:02.544506 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.544477    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.545716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.544687    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fcbt\" (UniqueName: \"kubernetes.io/projected/70adf136-b28a-439e-b53f-afce743b4a79-kube-api-access-8fcbt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.545716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.544711    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.545716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.544728    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.545716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.544743    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70adf136-b28a-439e-b53f-afce743b4a79-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.550402 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.550333    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-dshm" pod "70adf136-b28a-439e-b53f-afce743b4a79" (UID: "70adf136-b28a-439e-b53f-afce743b4a79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:02.557909 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.557884    5829 scope.go:122] "RemoveContainer" containerID="4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7"
Oct 31 07:45:02.567343 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:02.563356    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7\": container with ID starting with 4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7 not found: ID does not exist" containerID="4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7"
Oct 31 07:45:02.567343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.563403    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7"} err="failed to get container status \"4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7\": rpc error: code = NotFound desc = could not find container \"4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7\": container with ID starting with 4b841e3ed0354e435976ffdc404ab2fe5c14c95d0a37b6f80e74a1fa69f634c7 not found: ID does not exist"
Oct 31 07:45:02.567343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.563432    5829 scope.go:122] "RemoveContainer" containerID="19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24"
Oct 31 07:45:02.597723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.597697    5829 scope.go:122] "RemoveContainer" containerID="19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24"
Oct 31 07:45:02.598186 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:02.598154    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24\": container with ID starting with 19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24 not found: ID does not exist" containerID="19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24"
Oct 31 07:45:02.598238 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.598201    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24"} err="failed to get container status \"19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24\": rpc error: code = NotFound desc = could not find container \"19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24\": container with ID starting with 19428af0b9d0c7b82dc3afda6a92f881571144652ec9d9673d94e83bdeaedb24 not found: ID does not exist"
Oct 31 07:45:02.648827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.647698    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70adf136-b28a-439e-b53f-afce743b4a79-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:02.704359 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.704323    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"]
Oct 31 07:45:02.726332 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.723741    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/premium-w5-b1fda4-kserve-676cdf5d9f-rj2mb"]
Oct 31 07:45:02.863564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.863540    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:45:02.900851 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.900805    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"]
Oct 31 07:45:02.909631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.909599    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t"]
Oct 31 07:45:02.970661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.970618    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75a8cf6-1560-4982-9c99-6d59f996512b-kube-api-access-dcvvh\" (UniqueName: \"kubernetes.io/projected/b75a8cf6-1560-4982-9c99-6d59f996512b-kube-api-access-dcvvh\") pod \"b75a8cf6-1560-4982-9c99-6d59f996512b\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") "
Oct 31 07:45:02.970845 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.970686    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-openshift-ca\") pod \"b75a8cf6-1560-4982-9c99-6d59f996512b\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") "
Oct 31 07:45:02.971038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.971006    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-openshift-ca" pod "b75a8cf6-1560-4982-9c99-6d59f996512b" (UID: "b75a8cf6-1560-4982-9c99-6d59f996512b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:02.971240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.971212    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-xks-ca\") pod \"b75a8cf6-1560-4982-9c99-6d59f996512b\" (UID: \"b75a8cf6-1560-4982-9c99-6d59f996512b\") "
Oct 31 07:45:02.972152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:02.972129    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:03.004355 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.004229    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-xks-ca" pod "b75a8cf6-1560-4982-9c99-6d59f996512b" (UID: "b75a8cf6-1560-4982-9c99-6d59f996512b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:03.004727 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.004684    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75a8cf6-1560-4982-9c99-6d59f996512b-kube-api-access-dcvvh" pod "b75a8cf6-1560-4982-9c99-6d59f996512b" (UID: "b75a8cf6-1560-4982-9c99-6d59f996512b"). InnerVolumeSpecName "kube-api-access-dcvvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:03.073377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.073277    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcvvh\" (UniqueName: \"kubernetes.io/projected/b75a8cf6-1560-4982-9c99-6d59f996512b-kube-api-access-dcvvh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:03.073377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.073322    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75a8cf6-1560-4982-9c99-6d59f996512b-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:03.251605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.251550    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-b1fda4/simulator-w5-b1fda4-kserve-6fd979ccd6-fld2t" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:03.511489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.511447    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" event={"ID":"8965a35f-4b30-4e02-b22f-62634ae83b31","Type":"ContainerStarted","Data":"7e433576092b3a2d2601da657f035fc908213aad448e42c71f7e9c083a55f7df"}
Oct 31 07:45:03.514329 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.514269    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"
Oct 31 07:45:03.514493 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.514268    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w" event={"ID":"b75a8cf6-1560-4982-9c99-6d59f996512b","Type":"ContainerDied","Data":"7bcc70cf6c863bde31c4380612f561096162b36ab191f5c81ff8a0e4d63c7160"}
Oct 31 07:45:03.514493 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.514431    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bcc70cf6c863bde31c4380612f561096162b36ab191f5c81ff8a0e4d63c7160"
Oct 31 07:45:03.518142 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.518114    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" event={"ID":"d35962de-8dfc-477e-aebe-c0fb2ec68e5e","Type":"ContainerStarted","Data":"9e65a2cb33a5680b793b3fd30a81ff2d7553eaae5072bd35806e2dd24c7c01b3"}
Oct 31 07:45:03.520488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.520461    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" event={"ID":"993617de-c2f4-4f68-899b-f4cd1b192429","Type":"ContainerStarted","Data":"053da8b3696d452ac71624d52e5a2ade5b4451757740339a749dc34e98eb34d5"}
Oct 31 07:45:03.522428 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.522401    5829 generic.go:356] "Generic (PLEG): container finished" podID="0a6519d7-c1dc-40e4-b83a-a8a07670662c" containerID="65009fc9891eca7e1cc6522290b9bef9b8211fd6f59f38064f18465a2d76fc85" exitCode=0
Oct 31 07:45:03.522524 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.522440    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h" event={"ID":"0a6519d7-c1dc-40e4-b83a-a8a07670662c","Type":"ContainerDied","Data":"65009fc9891eca7e1cc6522290b9bef9b8211fd6f59f38064f18465a2d76fc85"}
Oct 31 07:45:03.555982 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.555917    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" podStartSLOduration=3.555903047 podStartE2EDuration="3.555903047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:03.551419156 +0000 UTC m=+967.690579743" watchObservedRunningTime="2026-10-31 07:45:03.555903047 +0000 UTC m=+967.695063634"
Oct 31 07:45:03.980898 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.980852    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0325aeb1-f4be-4a94-a3c2-c08937d33c00" path="/var/lib/kubelet/pods/0325aeb1-f4be-4a94-a3c2-c08937d33c00/volumes"
Oct 31 07:45:03.981559 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.981537    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70adf136-b28a-439e-b53f-afce743b4a79" path="/var/lib/kubelet/pods/70adf136-b28a-439e-b53f-afce743b4a79/volumes"
Oct 31 07:45:03.982024 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:03.982002    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" path="/var/lib/kubelet/pods/dc39e212-ced0-4425-a28e-e38ebfc09652/volumes"
Oct 31 07:45:04.542956 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.542915    5829 generic.go:356] "Generic (PLEG): container finished" podID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerID="9e65a2cb33a5680b793b3fd30a81ff2d7553eaae5072bd35806e2dd24c7c01b3" exitCode=60
Oct 31 07:45:04.543172 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.543004    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" event={"ID":"d35962de-8dfc-477e-aebe-c0fb2ec68e5e","Type":"ContainerDied","Data":"9e65a2cb33a5680b793b3fd30a81ff2d7553eaae5072bd35806e2dd24c7c01b3"}
Oct 31 07:45:04.543172 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.543091    5829 scope.go:122] "RemoveContainer" containerID="9dc55395d1909318a044edebcd5805aef65d0e8768bc960bd948e3a5ad163639"
Oct 31 07:45:04.543478 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.543457    5829 scope.go:122] "RemoveContainer" containerID="9e65a2cb33a5680b793b3fd30a81ff2d7553eaae5072bd35806e2dd24c7c01b3"
Oct 31 07:45:04.543983 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:04.543905    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv_odh-ai-gateway-infra(d35962de-8dfc-477e-aebe-c0fb2ec68e5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e"
Oct 31 07:45:04.548041 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.547214    5829 generic.go:356] "Generic (PLEG): container finished" podID="993617de-c2f4-4f68-899b-f4cd1b192429" containerID="053da8b3696d452ac71624d52e5a2ade5b4451757740339a749dc34e98eb34d5" exitCode=60
Oct 31 07:45:04.549049 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.548307    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" event={"ID":"993617de-c2f4-4f68-899b-f4cd1b192429","Type":"ContainerDied","Data":"053da8b3696d452ac71624d52e5a2ade5b4451757740339a749dc34e98eb34d5"}
Oct 31 07:45:04.549443 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.548797    5829 scope.go:122] "RemoveContainer" containerID="053da8b3696d452ac71624d52e5a2ade5b4451757740339a749dc34e98eb34d5"
Oct 31 07:45:04.555044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.555004    5829 generic.go:356] "Generic (PLEG): container finished" podID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerID="7e433576092b3a2d2601da657f035fc908213aad448e42c71f7e9c083a55f7df" exitCode=60
Oct 31 07:45:04.555820 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:04.555787    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd_odh-ai-gateway-infra(993617de-c2f4-4f68-899b-f4cd1b192429)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" podUID="993617de-c2f4-4f68-899b-f4cd1b192429"
Oct 31 07:45:04.555944 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.555876    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" event={"ID":"8965a35f-4b30-4e02-b22f-62634ae83b31","Type":"ContainerDied","Data":"7e433576092b3a2d2601da657f035fc908213aad448e42c71f7e9c083a55f7df"}
Oct 31 07:45:04.555944 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.555886    5829 scope.go:122] "RemoveContainer" containerID="7e433576092b3a2d2601da657f035fc908213aad448e42c71f7e9c083a55f7df"
Oct 31 07:45:04.558386 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:04.556364    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl_odh-ai-gateway-infra(8965a35f-4b30-4e02-b22f-62634ae83b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31"
Oct 31 07:45:04.567780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.567758    5829 scope.go:122] "RemoveContainer" containerID="3027478f8cebf23ecb70a55e95d76a783b835b92d7bc3c9a156fd53d1918d634"
Oct 31 07:45:04.585037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.585009    5829 scope.go:122] "RemoveContainer" containerID="d7966bb2aa911cd673fcd91a6ebf5d7f6c153553a275c5bab4808ae1e37e55f5"
Oct 31 07:45:04.883135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:04.883104    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:05.049513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.049477    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-openshift-ca\") pod \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") "
Oct 31 07:45:05.049973 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.049624    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6519d7-c1dc-40e4-b83a-a8a07670662c-kube-api-access-96mrq\" (UniqueName: \"kubernetes.io/projected/0a6519d7-c1dc-40e4-b83a-a8a07670662c-kube-api-access-96mrq\") pod \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") "
Oct 31 07:45:05.049973 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.049863    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-xks-ca\") pod \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\" (UID: \"0a6519d7-c1dc-40e4-b83a-a8a07670662c\") "
Oct 31 07:45:05.049973 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.049922    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-openshift-ca" pod "0a6519d7-c1dc-40e4-b83a-a8a07670662c" (UID: "0a6519d7-c1dc-40e4-b83a-a8a07670662c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:05.050713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.050692    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:05.052784 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.052754    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-xks-ca" pod "0a6519d7-c1dc-40e4-b83a-a8a07670662c" (UID: "0a6519d7-c1dc-40e4-b83a-a8a07670662c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:05.052909 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.052798    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6519d7-c1dc-40e4-b83a-a8a07670662c-kube-api-access-96mrq" pod "0a6519d7-c1dc-40e4-b83a-a8a07670662c" (UID: "0a6519d7-c1dc-40e4-b83a-a8a07670662c"). InnerVolumeSpecName "kube-api-access-96mrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:05.152004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.151958    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96mrq\" (UniqueName: \"kubernetes.io/projected/0a6519d7-c1dc-40e4-b83a-a8a07670662c-kube-api-access-96mrq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:05.152004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.151997    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a6519d7-c1dc-40e4-b83a-a8a07670662c-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:05.566513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.566419    5829 scope.go:122] "RemoveContainer" containerID="7e433576092b3a2d2601da657f035fc908213aad448e42c71f7e9c083a55f7df"
Oct 31 07:45:05.566832 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:05.566791    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl_odh-ai-gateway-infra(8965a35f-4b30-4e02-b22f-62634ae83b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31"
Oct 31 07:45:05.568447 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.568421    5829 scope.go:122] "RemoveContainer" containerID="9e65a2cb33a5680b793b3fd30a81ff2d7553eaae5072bd35806e2dd24c7c01b3"
Oct 31 07:45:05.568773 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:05.568741    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv_odh-ai-gateway-infra(d35962de-8dfc-477e-aebe-c0fb2ec68e5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e"
Oct 31 07:45:05.570063 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.570043    5829 scope.go:122] "RemoveContainer" containerID="053da8b3696d452ac71624d52e5a2ade5b4451757740339a749dc34e98eb34d5"
Oct 31 07:45:05.570403 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:05.570380    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd_odh-ai-gateway-infra(993617de-c2f4-4f68-899b-f4cd1b192429)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" podUID="993617de-c2f4-4f68-899b-f4cd1b192429"
Oct 31 07:45:05.571133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.571115    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"
Oct 31 07:45:05.571214 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.571117    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h" event={"ID":"0a6519d7-c1dc-40e4-b83a-a8a07670662c","Type":"ContainerDied","Data":"f4506d72ffa11c5a3d0057d9aa9b033f74a89e94dc220ad3c1947f8fea3e28d4"}
Oct 31 07:45:05.571291 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:05.571231    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4506d72ffa11c5a3d0057d9aa9b033f74a89e94dc220ad3c1947f8fea3e28d4"
Oct 31 07:45:08.002345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:08.002309    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"]
Oct 31 07:45:09.365238 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:09.361739    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"]
Oct 31 07:45:09.365238 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:09.361987    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" containerID="cri-o://ccc51f9f825c48df168bd98230a9cd0f059007724033926cc9bb4645e269f55c" gracePeriod=30
Oct 31 07:45:09.382983 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:09.382955    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"]
Oct 31 07:45:09.383202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:09.383178    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy" containerID="cri-o://92e9636f6fade172d695b32ffbcb6f7fff3d33f19df4979c01c2b6a78bce757d" gracePeriod=30
Oct 31 07:45:09.390349 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:09.390319    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"]
Oct 31 07:45:09.390570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:09.390545    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy" containerID="cri-o://2f6fad6bf73af5c29a2dd7843f9438d618605570f3bfb73dbb16b28d7c254dfc" gracePeriod=30
Oct 31 07:45:10.914462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:10.914421    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726" podUID="8f6c6f59-6834-4d62-9e21-1f263fff5b7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:10.914462 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:10.914448    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f" podUID="91c0428e-b966-4e6e-8425-15bc33a040b6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:11.169780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.169697    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"]
Oct 31 07:45:11.170490 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170473    5829 cpu_manager.go:414] "containerMap: removing container" podUID="b75a8cf6-1560-4982-9c99-6d59f996512b" containerName="revoke-keys"
Oct 31 07:45:11.170555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170491    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75a8cf6-1560-4982-9c99-6d59f996512b" containerName="revoke-keys"
Oct 31 07:45:11.170555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170515    5829 cpu_manager.go:414] "containerMap: removing container" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main"
Oct 31 07:45:11.170555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170521    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main"
Oct 31 07:45:11.170555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170533    5829 cpu_manager.go:414] "containerMap: removing container" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main"
Oct 31 07:45:11.170555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170538    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main"
Oct 31 07:45:11.170555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170554    5829 cpu_manager.go:414] "containerMap: removing container" podUID="0a6519d7-c1dc-40e4-b83a-a8a07670662c" containerName="revoke-keys"
Oct 31 07:45:11.170742 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170559    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6519d7-c1dc-40e4-b83a-a8a07670662c" containerName="revoke-keys"
Oct 31 07:45:11.170742 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170660    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6519d7-c1dc-40e4-b83a-a8a07670662c" containerName="revoke-keys"
Oct 31 07:45:11.170742 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170676    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75a8cf6-1560-4982-9c99-6d59f996512b" containerName="revoke-keys"
Oct 31 07:45:11.170742 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170685    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc39e212-ced0-4425-a28e-e38ebfc09652" containerName="main"
Oct 31 07:45:11.170742 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.170691    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="70adf136-b28a-439e-b53f-afce743b4a79" containerName="main"
Oct 31 07:45:11.176013 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.175988    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.192247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.192218    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"]
Oct 31 07:45:11.203965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.203930    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:45:11.305187 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.305144    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ffdd5d6-wwz4g\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.305501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.305478    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl6rs\" (UniqueName: \"kubernetes.io/projected/b17a2d17-acad-40a8-8c19-b1c641832ed9-kube-api-access-cl6rs\") pod \"maas-api-revoke-sub-4ffdd5d6-wwz4g\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.305612 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.305590    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ffdd5d6-wwz4g\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.407468 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.407424    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ffdd5d6-wwz4g\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.407900 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.407875    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ffdd5d6-wwz4g\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.408234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.408215    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl6rs\" (UniqueName: \"kubernetes.io/projected/b17a2d17-acad-40a8-8c19-b1c641832ed9-kube-api-access-cl6rs\") pod \"maas-api-revoke-sub-4ffdd5d6-wwz4g\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.417939 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.417902    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ffdd5d6-wwz4g\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.419817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.419678    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ffdd5d6-wwz4g\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.429075 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.429047    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl6rs\" (UniqueName: \"kubernetes.io/projected/b17a2d17-acad-40a8-8c19-b1c641832ed9-kube-api-access-cl6rs\") pod \"maas-api-revoke-sub-4ffdd5d6-wwz4g\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.491528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.491495    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:11.836060 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:45:11.836029    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17a2d17_acad_40a8_8c19_b1c641832ed9.slice/crio-81a46f443e40234f3e90ecaa279ba0c06d23972fb0961a47010b85611db1e784 WatchSource:0}: Error finding container 81a46f443e40234f3e90ecaa279ba0c06d23972fb0961a47010b85611db1e784: Status 404 returned error can't find the container with id 81a46f443e40234f3e90ecaa279ba0c06d23972fb0961a47010b85611db1e784
Oct 31 07:45:11.837073 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:11.837043    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"]
Oct 31 07:45:12.323512 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:12.323473    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 07:45:12.323961 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:12.323538    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:45:12.655907 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:12.655863    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g" event={"ID":"b17a2d17-acad-40a8-8c19-b1c641832ed9","Type":"ContainerStarted","Data":"f90d365297092dc030152e04ba1dcf5a7e1f2270976ab82e617604512bf71482"}
Oct 31 07:45:12.656113 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:12.655929    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g" event={"ID":"b17a2d17-acad-40a8-8c19-b1c641832ed9","Type":"ContainerStarted","Data":"81a46f443e40234f3e90ecaa279ba0c06d23972fb0961a47010b85611db1e784"}
Oct 31 07:45:12.696802 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:12.696748    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g" podStartSLOduration=1.696732417 podStartE2EDuration="1.696732417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:12.695020959 +0000 UTC m=+976.834181548" watchObservedRunningTime="2026-10-31 07:45:12.696732417 +0000 UTC m=+976.835893032"
Oct 31 07:45:13.003629 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:13.003533    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr" podUID="df54b1a3-b0e3-4481-b483-250b1136c78d" containerName="payload-processing" containerID="cri-o://b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3" gracePeriod=25
Oct 31 07:45:13.667467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:13.667424    5829 generic.go:356] "Generic (PLEG): container finished" podID="b17a2d17-acad-40a8-8c19-b1c641832ed9" containerID="f90d365297092dc030152e04ba1dcf5a7e1f2270976ab82e617604512bf71482" exitCode=0
Oct 31 07:45:13.667857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:13.667464    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g" event={"ID":"b17a2d17-acad-40a8-8c19-b1c641832ed9","Type":"ContainerDied","Data":"f90d365297092dc030152e04ba1dcf5a7e1f2270976ab82e617604512bf71482"}
Oct 31 07:45:14.144950 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.144912    5829 patch_prober.go:29] interesting pod/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 07:45:14.145123 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.144974    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:45:14.696314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.696262    5829 generic.go:356] "Generic (PLEG): container finished" podID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerID="92e9636f6fade172d695b32ffbcb6f7fff3d33f19df4979c01c2b6a78bce757d" exitCode=0
Oct 31 07:45:14.698207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.696585    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" event={"ID":"6229c3a0-d54b-49ac-9255-7c9d9594774a","Type":"ContainerDied","Data":"92e9636f6fade172d695b32ffbcb6f7fff3d33f19df4979c01c2b6a78bce757d"}
Oct 31 07:45:14.698207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.697483    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp" event={"ID":"6229c3a0-d54b-49ac-9255-7c9d9594774a","Type":"ContainerDied","Data":"baa789c5b64273d32cf6cbab698443b6d11cae66d5c0cc6724298b5e83813601"}
Oct 31 07:45:14.698207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.697546    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa789c5b64273d32cf6cbab698443b6d11cae66d5c0cc6724298b5e83813601"
Oct 31 07:45:14.704440 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.704407    5829 generic.go:356] "Generic (PLEG): container finished" podID="8b86fad7-2774-4996-9022-532c72f7e325" containerID="ccc51f9f825c48df168bd98230a9cd0f059007724033926cc9bb4645e269f55c" exitCode=0
Oct 31 07:45:14.704562 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.704485    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" event={"ID":"8b86fad7-2774-4996-9022-532c72f7e325","Type":"ContainerDied","Data":"ccc51f9f825c48df168bd98230a9cd0f059007724033926cc9bb4645e269f55c"}
Oct 31 07:45:14.707099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.707075    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:45:14.707473 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.707442    5829 generic.go:356] "Generic (PLEG): container finished" podID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerID="2f6fad6bf73af5c29a2dd7843f9438d618605570f3bfb73dbb16b28d7c254dfc" exitCode=0
Oct 31 07:45:14.707701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.707682    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" event={"ID":"d130a1c7-6b3f-4168-af6a-451147ed223d","Type":"ContainerDied","Data":"2f6fad6bf73af5c29a2dd7843f9438d618605570f3bfb73dbb16b28d7c254dfc"}
Oct 31 07:45:14.707787 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.707741    5829 scope.go:122] "RemoveContainer" containerID="4ab8a0adf6b0a59509566d296e59ce0e87751018e803a6cf57b5723a48ccdd39"
Oct 31 07:45:14.756253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.755744    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:45:14.764480 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.764448    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:45:14.874235 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874196    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d130a1c7-6b3f-4168-af6a-451147ed223d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d130a1c7-6b3f-4168-af6a-451147ed223d-istiod-ca-cert\") pod \"d130a1c7-6b3f-4168-af6a-451147ed223d\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") "
Oct 31 07:45:14.874482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874257    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-kube-api-access-mt9jj\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-kube-api-access-mt9jj\") pod \"8b86fad7-2774-4996-9022-532c72f7e325\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") "
Oct 31 07:45:14.874482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874373    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6229c3a0-d54b-49ac-9255-7c9d9594774a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6229c3a0-d54b-49ac-9255-7c9d9594774a-istiod-ca-cert\") pod \"6229c3a0-d54b-49ac-9255-7c9d9594774a\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") "
Oct 31 07:45:14.874482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874436    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-certs\") pod \"6229c3a0-d54b-49ac-9255-7c9d9594774a\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") "
Oct 31 07:45:14.874482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874472    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b86fad7-2774-4996-9022-532c72f7e325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b86fad7-2774-4996-9022-532c72f7e325-istio-podinfo\") pod \"8b86fad7-2774-4996-9022-532c72f7e325\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") "
Oct 31 07:45:14.874744 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874552    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-data\") pod \"d130a1c7-6b3f-4168-af6a-451147ed223d\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") "
Oct 31 07:45:14.874744 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874601    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-kube-api-access-xc8nd\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-kube-api-access-xc8nd\") pod \"d130a1c7-6b3f-4168-af6a-451147ed223d\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") "
Oct 31 07:45:14.874744 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874632    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-credential-socket\") pod \"d130a1c7-6b3f-4168-af6a-451147ed223d\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") "
Oct 31 07:45:14.874744 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874662    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-credential-socket\") pod \"6229c3a0-d54b-49ac-9255-7c9d9594774a\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") "
Oct 31 07:45:14.874744 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874739    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-credential-socket\") pod \"8b86fad7-2774-4996-9022-532c72f7e325\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") "
Oct 31 07:45:14.875002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874772    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-kube-api-access-clph7\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-kube-api-access-clph7\") pod \"6229c3a0-d54b-49ac-9255-7c9d9594774a\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") "
Oct 31 07:45:14.875002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874835    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-data\") pod \"6229c3a0-d54b-49ac-9255-7c9d9594774a\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") "
Oct 31 07:45:14.875002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874901    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-socket\") pod \"8b86fad7-2774-4996-9022-532c72f7e325\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") "
Oct 31 07:45:14.875002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874932    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-certs\") pod \"8b86fad7-2774-4996-9022-532c72f7e325\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") "
Oct 31 07:45:14.875002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.874988    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-socket\") pod \"6229c3a0-d54b-49ac-9255-7c9d9594774a\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") "
Oct 31 07:45:14.875326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875038    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-envoy\") pod \"d130a1c7-6b3f-4168-af6a-451147ed223d\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") "
Oct 31 07:45:14.875326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875099    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-podinfo\") pod \"6229c3a0-d54b-49ac-9255-7c9d9594774a\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") "
Oct 31 07:45:14.875326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875136    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-istio-token\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-istio-token\") pod \"8b86fad7-2774-4996-9022-532c72f7e325\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") "
Oct 31 07:45:14.875326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875220    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-token\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-token\") pod \"d130a1c7-6b3f-4168-af6a-451147ed223d\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") "
Oct 31 07:45:14.875326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875253    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-certs\") pod \"d130a1c7-6b3f-4168-af6a-451147ed223d\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") "
Oct 31 07:45:14.875326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875302    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-token\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-token\") pod \"6229c3a0-d54b-49ac-9255-7c9d9594774a\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") "
Oct 31 07:45:14.875631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875370    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-data\") pod \"8b86fad7-2774-4996-9022-532c72f7e325\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") "
Oct 31 07:45:14.875631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875439    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-envoy\") pod \"8b86fad7-2774-4996-9022-532c72f7e325\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") "
Oct 31 07:45:14.875631 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875484    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-podinfo\") pod \"d130a1c7-6b3f-4168-af6a-451147ed223d\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") "
Oct 31 07:45:14.875815 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875794    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-envoy\") pod \"6229c3a0-d54b-49ac-9255-7c9d9594774a\" (UID: \"6229c3a0-d54b-49ac-9255-7c9d9594774a\") "
Oct 31 07:45:14.875879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875862    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b86fad7-2774-4996-9022-532c72f7e325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b86fad7-2774-4996-9022-532c72f7e325-istiod-ca-cert\") pod \"8b86fad7-2774-4996-9022-532c72f7e325\" (UID: \"8b86fad7-2774-4996-9022-532c72f7e325\") "
Oct 31 07:45:14.875933 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.875893    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-socket\") pod \"d130a1c7-6b3f-4168-af6a-451147ed223d\" (UID: \"d130a1c7-6b3f-4168-af6a-451147ed223d\") "
Oct 31 07:45:14.876971 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.876940    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-data" pod "6229c3a0-d54b-49ac-9255-7c9d9594774a" (UID: "6229c3a0-d54b-49ac-9255-7c9d9594774a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.878101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.877270    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d130a1c7-6b3f-4168-af6a-451147ed223d-istiod-ca-cert" pod "d130a1c7-6b3f-4168-af6a-451147ed223d" (UID: "d130a1c7-6b3f-4168-af6a-451147ed223d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:14.878101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.877654    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-socket" pod "d130a1c7-6b3f-4168-af6a-451147ed223d" (UID: "d130a1c7-6b3f-4168-af6a-451147ed223d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.878101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.877997    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-socket" pod "8b86fad7-2774-4996-9022-532c72f7e325" (UID: "8b86fad7-2774-4996-9022-532c72f7e325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.878383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.878163    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-certs" pod "8b86fad7-2774-4996-9022-532c72f7e325" (UID: "8b86fad7-2774-4996-9022-532c72f7e325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.878383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.878344    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-socket" pod "6229c3a0-d54b-49ac-9255-7c9d9594774a" (UID: "6229c3a0-d54b-49ac-9255-7c9d9594774a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.879061 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.879032    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-credential-socket" pod "8b86fad7-2774-4996-9022-532c72f7e325" (UID: "8b86fad7-2774-4996-9022-532c72f7e325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.879269 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.879241    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-credential-socket" pod "6229c3a0-d54b-49ac-9255-7c9d9594774a" (UID: "6229c3a0-d54b-49ac-9255-7c9d9594774a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.879533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.879504    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-certs" pod "6229c3a0-d54b-49ac-9255-7c9d9594774a" (UID: "6229c3a0-d54b-49ac-9255-7c9d9594774a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.880337 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.880049    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-credential-socket" pod "d130a1c7-6b3f-4168-af6a-451147ed223d" (UID: "d130a1c7-6b3f-4168-af6a-451147ed223d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.880337 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.880111    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-kube-api-access-mt9jj" pod "8b86fad7-2774-4996-9022-532c72f7e325" (UID: "8b86fad7-2774-4996-9022-532c72f7e325"). InnerVolumeSpecName "kube-api-access-mt9jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:14.880961 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.880664    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6229c3a0-d54b-49ac-9255-7c9d9594774a-istiod-ca-cert" pod "6229c3a0-d54b-49ac-9255-7c9d9594774a" (UID: "6229c3a0-d54b-49ac-9255-7c9d9594774a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:14.882640 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.882612    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-istio-token" pod "8b86fad7-2774-4996-9022-532c72f7e325" (UID: "8b86fad7-2774-4996-9022-532c72f7e325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:14.882744 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.882683    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-envoy" pod "d130a1c7-6b3f-4168-af6a-451147ed223d" (UID: "d130a1c7-6b3f-4168-af6a-451147ed223d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.882912 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.882893    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-data" pod "d130a1c7-6b3f-4168-af6a-451147ed223d" (UID: "d130a1c7-6b3f-4168-af6a-451147ed223d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.883207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.883184    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b86fad7-2774-4996-9022-532c72f7e325-istiod-ca-cert" pod "8b86fad7-2774-4996-9022-532c72f7e325" (UID: "8b86fad7-2774-4996-9022-532c72f7e325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:14.883498 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.883472    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-data" pod "8b86fad7-2774-4996-9022-532c72f7e325" (UID: "8b86fad7-2774-4996-9022-532c72f7e325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.883588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.883528    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-kube-api-access-xc8nd" pod "d130a1c7-6b3f-4168-af6a-451147ed223d" (UID: "d130a1c7-6b3f-4168-af6a-451147ed223d"). InnerVolumeSpecName "kube-api-access-xc8nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:14.884038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.883785    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-certs" pod "d130a1c7-6b3f-4168-af6a-451147ed223d" (UID: "d130a1c7-6b3f-4168-af6a-451147ed223d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.884176 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.884146    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-podinfo" pod "d130a1c7-6b3f-4168-af6a-451147ed223d" (UID: "d130a1c7-6b3f-4168-af6a-451147ed223d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:14.884637 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.884588    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-podinfo" pod "6229c3a0-d54b-49ac-9255-7c9d9594774a" (UID: "6229c3a0-d54b-49ac-9255-7c9d9594774a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:14.885236 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.885197    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-envoy" pod "6229c3a0-d54b-49ac-9255-7c9d9594774a" (UID: "6229c3a0-d54b-49ac-9255-7c9d9594774a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.885486 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.885458    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-token" pod "6229c3a0-d54b-49ac-9255-7c9d9594774a" (UID: "6229c3a0-d54b-49ac-9255-7c9d9594774a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:14.886232 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.886210    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-token" pod "d130a1c7-6b3f-4168-af6a-451147ed223d" (UID: "d130a1c7-6b3f-4168-af6a-451147ed223d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:14.886387 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.886367    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-envoy" pod "8b86fad7-2774-4996-9022-532c72f7e325" (UID: "8b86fad7-2774-4996-9022-532c72f7e325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:14.886450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.886392    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b86fad7-2774-4996-9022-532c72f7e325-istio-podinfo" pod "8b86fad7-2774-4996-9022-532c72f7e325" (UID: "8b86fad7-2774-4996-9022-532c72f7e325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:14.891560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.891533    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-kube-api-access-clph7" pod "6229c3a0-d54b-49ac-9255-7c9d9594774a" (UID: "6229c3a0-d54b-49ac-9255-7c9d9594774a"). InnerVolumeSpecName "kube-api-access-clph7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:14.913333 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.913304    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:14.967544 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.967508    5829 scope.go:122] "RemoveContainer" containerID="7e433576092b3a2d2601da657f035fc908213aad448e42c71f7e9c083a55f7df"
Oct 31 07:45:14.967728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.967586    5829 scope.go:122] "RemoveContainer" containerID="9e65a2cb33a5680b793b3fd30a81ff2d7553eaae5072bd35806e2dd24c7c01b3"
Oct 31 07:45:14.978392 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978365    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978392 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978392    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978405    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978417    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978430    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978442    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978454    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978466    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978479    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b86fad7-2774-4996-9022-532c72f7e325-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978491    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978504    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d130a1c7-6b3f-4168-af6a-451147ed223d-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978525    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt9jj\" (UniqueName: \"kubernetes.io/projected/8b86fad7-2774-4996-9022-532c72f7e325-kube-api-access-mt9jj\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978540    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6229c3a0-d54b-49ac-9255-7c9d9594774a-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978553    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978566    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b86fad7-2774-4996-9022-532c72f7e325-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.978576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978578    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978592    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc8nd\" (UniqueName: \"kubernetes.io/projected/d130a1c7-6b3f-4168-af6a-451147ed223d-kube-api-access-xc8nd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978605    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978620    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978633    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978647    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clph7\" (UniqueName: \"kubernetes.io/projected/6229c3a0-d54b-49ac-9255-7c9d9594774a-kube-api-access-clph7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978662    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978678    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978691    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b86fad7-2774-4996-9022-532c72f7e325-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978704    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6229c3a0-d54b-49ac-9255-7c9d9594774a-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978719    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d130a1c7-6b3f-4168-af6a-451147ed223d-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:14.979044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:14.978732    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6229c3a0-d54b-49ac-9255-7c9d9594774a-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:15.079494 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.079454    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-xks-ca\") pod \"b17a2d17-acad-40a8-8c19-b1c641832ed9\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") "
Oct 31 07:45:15.079686 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.079578    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17a2d17-acad-40a8-8c19-b1c641832ed9-kube-api-access-cl6rs\" (UniqueName: \"kubernetes.io/projected/b17a2d17-acad-40a8-8c19-b1c641832ed9-kube-api-access-cl6rs\") pod \"b17a2d17-acad-40a8-8c19-b1c641832ed9\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") "
Oct 31 07:45:15.079847 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.079826    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-openshift-ca\") pod \"b17a2d17-acad-40a8-8c19-b1c641832ed9\" (UID: \"b17a2d17-acad-40a8-8c19-b1c641832ed9\") "
Oct 31 07:45:15.080245 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.080221    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-openshift-ca" pod "b17a2d17-acad-40a8-8c19-b1c641832ed9" (UID: "b17a2d17-acad-40a8-8c19-b1c641832ed9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:15.081407 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.081376    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:15.081938 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.081918    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-xks-ca" pod "b17a2d17-acad-40a8-8c19-b1c641832ed9" (UID: "b17a2d17-acad-40a8-8c19-b1c641832ed9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:15.082030 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.082008    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17a2d17-acad-40a8-8c19-b1c641832ed9-kube-api-access-cl6rs" pod "b17a2d17-acad-40a8-8c19-b1c641832ed9" (UID: "b17a2d17-acad-40a8-8c19-b1c641832ed9"). InnerVolumeSpecName "kube-api-access-cl6rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:15.098557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.098525    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"]
Oct 31 07:45:15.099384 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099360    5829 cpu_manager.go:414] "containerMap: removing container" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy"
Oct 31 07:45:15.099384 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099382    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy"
Oct 31 07:45:15.099520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099397    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy"
Oct 31 07:45:15.099520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099402    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy"
Oct 31 07:45:15.099520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099432    5829 cpu_manager.go:414] "containerMap: removing container" podUID="b17a2d17-acad-40a8-8c19-b1c641832ed9" containerName="revoke-keys"
Oct 31 07:45:15.099520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099438    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17a2d17-acad-40a8-8c19-b1c641832ed9" containerName="revoke-keys"
Oct 31 07:45:15.099520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099448    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy"
Oct 31 07:45:15.099520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099452    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy"
Oct 31 07:45:15.099696 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099578    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17a2d17-acad-40a8-8c19-b1c641832ed9" containerName="revoke-keys"
Oct 31 07:45:15.099696 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099586    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b86fad7-2774-4996-9022-532c72f7e325" containerName="istio-proxy"
Oct 31 07:45:15.099696 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099598    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy"
Oct 31 07:45:15.099696 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099606    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" containerName="istio-proxy"
Oct 31 07:45:15.099696 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.099613    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy"
Oct 31 07:45:15.106109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.105041    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.128104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.128067    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"]
Oct 31 07:45:15.183018 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.182934    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl6rs\" (UniqueName: \"kubernetes.io/projected/b17a2d17-acad-40a8-8c19-b1c641832ed9-kube-api-access-cl6rs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:15.183018 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.182966    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b17a2d17-acad-40a8-8c19-b1c641832ed9-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:15.283932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.283894    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6sbl\" (UniqueName: \"kubernetes.io/projected/8d035bbe-d467-4f01-8325-8ccac7e65551-kube-api-access-p6sbl\") pod \"maas-api-revoke-sub-7dd675ee-v2q49\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.284145 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.284046    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dd675ee-v2q49\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.284145 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.284115    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dd675ee-v2q49\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.385579 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.385541    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dd675ee-v2q49\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.385779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.385598    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dd675ee-v2q49\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.385879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.385863    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6sbl\" (UniqueName: \"kubernetes.io/projected/8d035bbe-d467-4f01-8325-8ccac7e65551-kube-api-access-p6sbl\") pod \"maas-api-revoke-sub-7dd675ee-v2q49\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.386326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.386273    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7dd675ee-v2q49\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.388058 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.388032    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7dd675ee-v2q49\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.405865 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.405841    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6sbl\" (UniqueName: \"kubernetes.io/projected/8d035bbe-d467-4f01-8325-8ccac7e65551-kube-api-access-p6sbl\") pod \"maas-api-revoke-sub-7dd675ee-v2q49\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.423610 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.423588    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:15.718628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.718537    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" event={"ID":"8965a35f-4b30-4e02-b22f-62634ae83b31","Type":"ContainerStarted","Data":"ef59c1eeaa736d29752ccc1273a9d233019f846584646d901fc6186d6c130001"}
Oct 31 07:45:15.720515 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.720485    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr" event={"ID":"8b86fad7-2774-4996-9022-532c72f7e325","Type":"ContainerDied","Data":"0522625a1e8807c8b10d237c5f9f39fea7b4d42e0c40432687ec5223426847c0"}
Oct 31 07:45:15.720676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.720516    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"
Oct 31 07:45:15.720676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.720550    5829 scope.go:122] "RemoveContainer" containerID="ccc51f9f825c48df168bd98230a9cd0f059007724033926cc9bb4645e269f55c"
Oct 31 07:45:15.722318 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.722255    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"
Oct 31 07:45:15.722414 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.722324    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb" event={"ID":"d130a1c7-6b3f-4168-af6a-451147ed223d","Type":"ContainerDied","Data":"272488492fcdc251af1386b3ad5e177214bc281a20b8721e04fd295c3e3e29bc"}
Oct 31 07:45:15.725763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.725143    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" event={"ID":"d35962de-8dfc-477e-aebe-c0fb2ec68e5e","Type":"ContainerStarted","Data":"bb3f3a556b561de0b9a54988c9ca907ddae4b802c9d41a06153a309d92b665d4"}
Oct 31 07:45:15.730112 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.729795    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"
Oct 31 07:45:15.730900 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.730505    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g" event={"ID":"b17a2d17-acad-40a8-8c19-b1c641832ed9","Type":"ContainerDied","Data":"81a46f443e40234f3e90ecaa279ba0c06d23972fb0961a47010b85611db1e784"}
Oct 31 07:45:15.730900 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.730564    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81a46f443e40234f3e90ecaa279ba0c06d23972fb0961a47010b85611db1e784"
Oct 31 07:45:15.732570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.732549    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"
Oct 31 07:45:15.742038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.742008    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"]
Oct 31 07:45:15.742495 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.742476    5829 scope.go:122] "RemoveContainer" containerID="2f6fad6bf73af5c29a2dd7843f9438d618605570f3bfb73dbb16b28d7c254dfc"
Oct 31 07:45:15.799416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.799379    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"]
Oct 31 07:45:15.804626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.804598    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-7nfmr"]
Oct 31 07:45:15.924857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.924829    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"]
Oct 31 07:45:15.929983 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.929961    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-slbrb"]
Oct 31 07:45:15.962032 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.961998    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"]
Oct 31 07:45:15.973106 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.973032    5829 scope.go:122] "RemoveContainer" containerID="053da8b3696d452ac71624d52e5a2ade5b4451757740339a749dc34e98eb34d5"
Oct 31 07:45:15.976558 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.976476    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b86fad7-2774-4996-9022-532c72f7e325" path="/var/lib/kubelet/pods/8b86fad7-2774-4996-9022-532c72f7e325/volumes"
Oct 31 07:45:15.976995 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.976980    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" path="/var/lib/kubelet/pods/d130a1c7-6b3f-4168-af6a-451147ed223d/volumes"
Oct 31 07:45:15.977352 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:15.977338    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-027d2a-openshift-default-849f6d6869-rnsrp"]
Oct 31 07:45:16.047877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.047841    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"]
Oct 31 07:45:16.053204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.053171    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"]
Oct 31 07:45:16.066433 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.066396    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"]
Oct 31 07:45:16.524373 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:16.524335    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:45:16.742232 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.742197    5829 generic.go:356] "Generic (PLEG): container finished" podID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerID="ef59c1eeaa736d29752ccc1273a9d233019f846584646d901fc6186d6c130001" exitCode=60
Oct 31 07:45:16.742699 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.742290    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" event={"ID":"8965a35f-4b30-4e02-b22f-62634ae83b31","Type":"ContainerDied","Data":"ef59c1eeaa736d29752ccc1273a9d233019f846584646d901fc6186d6c130001"}
Oct 31 07:45:16.742699 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.742384    5829 scope.go:122] "RemoveContainer" containerID="7e433576092b3a2d2601da657f035fc908213aad448e42c71f7e9c083a55f7df"
Oct 31 07:45:16.742896 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.742878    5829 scope.go:122] "RemoveContainer" containerID="ef59c1eeaa736d29752ccc1273a9d233019f846584646d901fc6186d6c130001"
Oct 31 07:45:16.743690 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:16.743309    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl_odh-ai-gateway-infra(8965a35f-4b30-4e02-b22f-62634ae83b31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31"
Oct 31 07:45:16.744363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.744339    5829 generic.go:356] "Generic (PLEG): container finished" podID="8d035bbe-d467-4f01-8325-8ccac7e65551" containerID="3dec4a91bc4b01564357edd76664fb743dae73bc2822f94d458d0967d84fbf92" exitCode=0
Oct 31 07:45:16.744473 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.744421    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49" event={"ID":"8d035bbe-d467-4f01-8325-8ccac7e65551","Type":"ContainerDied","Data":"3dec4a91bc4b01564357edd76664fb743dae73bc2822f94d458d0967d84fbf92"}
Oct 31 07:45:16.744537 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.744472    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49" event={"ID":"8d035bbe-d467-4f01-8325-8ccac7e65551","Type":"ContainerStarted","Data":"3e26a203da1a10cd2e87fee37f0fc95bba24f5157881e7a4b93608a97c99a9ab"}
Oct 31 07:45:16.749112 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.749085    5829 generic.go:356] "Generic (PLEG): container finished" podID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerID="bb3f3a556b561de0b9a54988c9ca907ddae4b802c9d41a06153a309d92b665d4" exitCode=60
Oct 31 07:45:16.749234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.749161    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" event={"ID":"d35962de-8dfc-477e-aebe-c0fb2ec68e5e","Type":"ContainerDied","Data":"bb3f3a556b561de0b9a54988c9ca907ddae4b802c9d41a06153a309d92b665d4"}
Oct 31 07:45:16.749709 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.749685    5829 scope.go:122] "RemoveContainer" containerID="bb3f3a556b561de0b9a54988c9ca907ddae4b802c9d41a06153a309d92b665d4"
Oct 31 07:45:16.750122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.750093    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main" containerID="cri-o://2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376" gracePeriod=45
Oct 31 07:45:16.750242 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:16.750147    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv_odh-ai-gateway-infra(d35962de-8dfc-477e-aebe-c0fb2ec68e5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e"
Oct 31 07:45:16.751996 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.751973    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main" containerID="cri-o://ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650" gracePeriod=45
Oct 31 07:45:16.752377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.752196    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" event={"ID":"993617de-c2f4-4f68-899b-f4cd1b192429","Type":"ContainerStarted","Data":"34292fdac8b270b13db5377d7aaedd14661648ac96565efec44df9c507a510fb"}
Oct 31 07:45:16.759302 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.759268    5829 scope.go:122] "RemoveContainer" containerID="9e65a2cb33a5680b793b3fd30a81ff2d7553eaae5072bd35806e2dd24c7c01b3"
Oct 31 07:45:16.772855 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.772814    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"]
Oct 31 07:45:16.868129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:16.868096    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"]
Oct 31 07:45:17.066987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.066955    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:45:17.081551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.081472    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:45:17.116787 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.116750    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-home\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-home\") pod \"549cac55-429e-46a1-b33d-a619f959a170\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") "
Oct 31 07:45:17.116787 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.116800    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/549cac55-429e-46a1-b33d-a619f959a170-tls-certs\" (UniqueName: \"kubernetes.io/secret/549cac55-429e-46a1-b33d-a619f959a170-tls-certs\") pod \"549cac55-429e-46a1-b33d-a619f959a170\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") "
Oct 31 07:45:17.117002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.116859    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-model-cache\") pod \"549cac55-429e-46a1-b33d-a619f959a170\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") "
Oct 31 07:45:17.117002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.116893    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-model-cache\") pod \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") "
Oct 31 07:45:17.117002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.116983    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-dshm\") pod \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") "
Oct 31 07:45:17.117112 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117022    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tls-certs\") pod \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") "
Oct 31 07:45:17.117112 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117065    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-home\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-home\") pod \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") "
Oct 31 07:45:17.117112 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117059    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-home" pod "549cac55-429e-46a1-b33d-a619f959a170" (UID: "549cac55-429e-46a1-b33d-a619f959a170"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:17.117229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117120    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-kube-api-access-8f79k\" (UniqueName: \"kubernetes.io/projected/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-kube-api-access-8f79k\") pod \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") "
Oct 31 07:45:17.117229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117165    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-dshm\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-dshm\") pod \"549cac55-429e-46a1-b33d-a619f959a170\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") "
Oct 31 07:45:17.117229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117209    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-tmp-dir\") pod \"549cac55-429e-46a1-b33d-a619f959a170\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") "
Oct 31 07:45:17.117402 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117262    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tmp-dir\") pod \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\" (UID: \"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5\") "
Oct 31 07:45:17.117402 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117268    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-model-cache" pod "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" (UID: "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:17.117402 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117338    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/549cac55-429e-46a1-b33d-a619f959a170-kube-api-access-qsp65\" (UniqueName: \"kubernetes.io/projected/549cac55-429e-46a1-b33d-a619f959a170-kube-api-access-qsp65\") pod \"549cac55-429e-46a1-b33d-a619f959a170\" (UID: \"549cac55-429e-46a1-b33d-a619f959a170\") "
Oct 31 07:45:17.118080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117789    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-home" pod "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" (UID: "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:17.118080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117794    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-model-cache" pod "549cac55-429e-46a1-b33d-a619f959a170" (UID: "549cac55-429e-46a1-b33d-a619f959a170"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:17.118080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117833    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-tmp-dir" pod "549cac55-429e-46a1-b33d-a619f959a170" (UID: "549cac55-429e-46a1-b33d-a619f959a170"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:17.118080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117949    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tmp-dir" pod "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" (UID: "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:17.118080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117960    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.118080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117981    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.118080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.117995    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.118080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.118006    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.118080 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.118017    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.124788 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.124757    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-kube-api-access-8f79k" pod "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" (UID: "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5"). InnerVolumeSpecName "kube-api-access-8f79k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:17.128566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.128516    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-dshm" pod "549cac55-429e-46a1-b33d-a619f959a170" (UID: "549cac55-429e-46a1-b33d-a619f959a170"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:17.128566 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.128530    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549cac55-429e-46a1-b33d-a619f959a170-kube-api-access-qsp65" pod "549cac55-429e-46a1-b33d-a619f959a170" (UID: "549cac55-429e-46a1-b33d-a619f959a170"). InnerVolumeSpecName "kube-api-access-qsp65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:17.128883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.128794    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549cac55-429e-46a1-b33d-a619f959a170-tls-certs" pod "549cac55-429e-46a1-b33d-a619f959a170" (UID: "549cac55-429e-46a1-b33d-a619f959a170"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:17.128883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.128833    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tls-certs" pod "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" (UID: "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:17.131348 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.130423    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-dshm" pod "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" (UID: "e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:17.219105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.219017    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.219105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.219048    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.219105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.219061    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f79k\" (UniqueName: \"kubernetes.io/projected/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-kube-api-access-8f79k\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.219105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.219077    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/549cac55-429e-46a1-b33d-a619f959a170-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.219105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.219089    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.219105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.219103    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsp65\" (UniqueName: \"kubernetes.io/projected/549cac55-429e-46a1-b33d-a619f959a170-kube-api-access-qsp65\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.219419 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.219116    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/549cac55-429e-46a1-b33d-a619f959a170-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:17.766948 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.766903    5829 generic.go:356] "Generic (PLEG): container finished" podID="8f6c6f59-6834-4d62-9e21-1f263fff5b7b" containerID="f78ba7606153235b5125ad56dea27f82241bc7bb195332807e8048cf797cc5b1" exitCode=0
Oct 31 07:45:17.767270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.766953    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726" event={"ID":"8f6c6f59-6834-4d62-9e21-1f263fff5b7b","Type":"ContainerDied","Data":"f78ba7606153235b5125ad56dea27f82241bc7bb195332807e8048cf797cc5b1"}
Oct 31 07:45:17.769324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.769293    5829 generic.go:356] "Generic (PLEG): container finished" podID="91c0428e-b966-4e6e-8425-15bc33a040b6" containerID="179a509c109305b296d36c07e45dd5fe48277e0ff7775fba63ff7d076bad7f47" exitCode=0
Oct 31 07:45:17.769324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.769310    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f" event={"ID":"91c0428e-b966-4e6e-8425-15bc33a040b6","Type":"ContainerDied","Data":"179a509c109305b296d36c07e45dd5fe48277e0ff7775fba63ff7d076bad7f47"}
Oct 31 07:45:17.771040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.771016    5829 generic.go:356] "Generic (PLEG): container finished" podID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerID="2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376" exitCode=0
Oct 31 07:45:17.771157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.771084    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"
Oct 31 07:45:17.771157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.771097    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" event={"ID":"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5","Type":"ContainerDied","Data":"2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376"}
Oct 31 07:45:17.771157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.771147    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c" event={"ID":"e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5","Type":"ContainerDied","Data":"d2973c47118767416b18a1d19b6601fa503f746acde92838d85052ff06218024"}
Oct 31 07:45:17.771387 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.771171    5829 scope.go:122] "RemoveContainer" containerID="2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376"
Oct 31 07:45:17.776053 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.776025    5829 generic.go:356] "Generic (PLEG): container finished" podID="993617de-c2f4-4f68-899b-f4cd1b192429" containerID="34292fdac8b270b13db5377d7aaedd14661648ac96565efec44df9c507a510fb" exitCode=60
Oct 31 07:45:17.776167 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.776126    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" event={"ID":"993617de-c2f4-4f68-899b-f4cd1b192429","Type":"ContainerDied","Data":"34292fdac8b270b13db5377d7aaedd14661648ac96565efec44df9c507a510fb"}
Oct 31 07:45:17.777033 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.776678    5829 scope.go:122] "RemoveContainer" containerID="34292fdac8b270b13db5377d7aaedd14661648ac96565efec44df9c507a510fb"
Oct 31 07:45:17.777225 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:17.777186    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd_odh-ai-gateway-infra(993617de-c2f4-4f68-899b-f4cd1b192429)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" podUID="993617de-c2f4-4f68-899b-f4cd1b192429"
Oct 31 07:45:17.784141 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.784117    5829 generic.go:356] "Generic (PLEG): container finished" podID="549cac55-429e-46a1-b33d-a619f959a170" containerID="ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650" exitCode=0
Oct 31 07:45:17.784270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.784188    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"
Oct 31 07:45:17.784270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.784194    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" event={"ID":"549cac55-429e-46a1-b33d-a619f959a170","Type":"ContainerDied","Data":"ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650"}
Oct 31 07:45:17.784270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.784243    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv" event={"ID":"549cac55-429e-46a1-b33d-a619f959a170","Type":"ContainerDied","Data":"a0444008303d41cf58ba5f1839c5df5288cb1b4202d15b35ee73db686c85c610"}
Oct 31 07:45:17.789688 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.789664    5829 scope.go:122] "RemoveContainer" containerID="2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376"
Oct 31 07:45:17.792416 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:17.792382    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376\": container with ID starting with 2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376 not found: ID does not exist" containerID="2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376"
Oct 31 07:45:17.792529 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.792417    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376"} err="failed to get container status \"2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376\": rpc error: code = NotFound desc = could not find container \"2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376\": container with ID starting with 2563b27d0a730062f79b93d3c9f1f03c5fcd033229f41b7810f43d569ddf1376 not found: ID does not exist"
Oct 31 07:45:17.792529 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.792435    5829 scope.go:122] "RemoveContainer" containerID="053da8b3696d452ac71624d52e5a2ade5b4451757740339a749dc34e98eb34d5"
Oct 31 07:45:17.821518 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.821486    5829 scope.go:122] "RemoveContainer" containerID="ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650"
Oct 31 07:45:17.860041 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.859108    5829 scope.go:122] "RemoveContainer" containerID="ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650"
Oct 31 07:45:17.860412 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:17.860303    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650\": container with ID starting with ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650 not found: ID does not exist" containerID="ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650"
Oct 31 07:45:17.860412 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.860347    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650"} err="failed to get container status \"ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650\": rpc error: code = NotFound desc = could not find container \"ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650\": container with ID starting with ff6f417be4884c6c8a02e4ffb756aedabd1a841970fac87d186fd7f73836e650 not found: ID does not exist"
Oct 31 07:45:17.890363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.889336    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr" podUID="df54b1a3-b0e3-4481-b483-250b1136c78d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:17.928816 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:17.928618    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"]
Oct 31 07:45:18.009173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.009140    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6229c3a0-d54b-49ac-9255-7c9d9594774a" path="/var/lib/kubelet/pods/6229c3a0-d54b-49ac-9255-7c9d9594774a/volumes"
Oct 31 07:45:18.009843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.009817    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"]
Oct 31 07:45:18.011129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.010501    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/distinct-2-w4-3dff3f-kserve-64b9878548-9qc8c"]
Oct 31 07:45:18.013811 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.013783    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:45:18.020567 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.020541    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"]
Oct 31 07:45:18.035970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.035940    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-plugins-config-volume\") pod \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\" (UID: \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\") "
Oct 31 07:45:18.036297 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.036258    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-kube-api-access-khjh8\" (UniqueName: \"kubernetes.io/projected/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-kube-api-access-khjh8\") pod \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\" (UID: \"8f6c6f59-6834-4d62-9e21-1f263fff5b7b\") "
Oct 31 07:45:18.036678 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.036652    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3dff3f/premium-w4-3dff3f-kserve-68fbd694dd-q7vlv"]
Oct 31 07:45:18.038163 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.038135    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-plugins-config-volume" pod "8f6c6f59-6834-4d62-9e21-1f263fff5b7b" (UID: "8f6c6f59-6834-4d62-9e21-1f263fff5b7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:18.042941 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.041217    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-kube-api-access-khjh8" pod "8f6c6f59-6834-4d62-9e21-1f263fff5b7b" (UID: "8f6c6f59-6834-4d62-9e21-1f263fff5b7b"). InnerVolumeSpecName "kube-api-access-khjh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:18.049925 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.049903    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:45:18.111427 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.111362    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:18.116820 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.116795    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:18.123965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.123935    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:18.142327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.142275    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d035bbe-d467-4f01-8325-8ccac7e65551-kube-api-access-p6sbl\" (UniqueName: \"kubernetes.io/projected/8d035bbe-d467-4f01-8325-8ccac7e65551-kube-api-access-p6sbl\") pod \"8d035bbe-d467-4f01-8325-8ccac7e65551\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") "
Oct 31 07:45:18.142495 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.142358    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8965a35f-4b30-4e02-b22f-62634ae83b31-kube-api-access-ljmzf\" (UniqueName: \"kubernetes.io/projected/8965a35f-4b30-4e02-b22f-62634ae83b31-kube-api-access-ljmzf\") pod \"8965a35f-4b30-4e02-b22f-62634ae83b31\" (UID: \"8965a35f-4b30-4e02-b22f-62634ae83b31\") "
Oct 31 07:45:18.142495 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.142419    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c0428e-b966-4e6e-8425-15bc33a040b6-kube-api-access-ddlnl\" (UniqueName: \"kubernetes.io/projected/91c0428e-b966-4e6e-8425-15bc33a040b6-kube-api-access-ddlnl\") pod \"91c0428e-b966-4e6e-8425-15bc33a040b6\" (UID: \"91c0428e-b966-4e6e-8425-15bc33a040b6\") "
Oct 31 07:45:18.142647 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.142536    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8965a35f-4b30-4e02-b22f-62634ae83b31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965a35f-4b30-4e02-b22f-62634ae83b31-maas-api-service-ca\") pod \"8965a35f-4b30-4e02-b22f-62634ae83b31\" (UID: \"8965a35f-4b30-4e02-b22f-62634ae83b31\") "
Oct 31 07:45:18.142647 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.142622    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-openshift-ca\") pod \"8d035bbe-d467-4f01-8325-8ccac7e65551\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") "
Oct 31 07:45:18.142756 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.142686    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c0428e-b966-4e6e-8425-15bc33a040b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c0428e-b966-4e6e-8425-15bc33a040b6-plugins-config-volume\") pod \"91c0428e-b966-4e6e-8425-15bc33a040b6\" (UID: \"91c0428e-b966-4e6e-8425-15bc33a040b6\") "
Oct 31 07:45:18.142756 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.142725    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-kube-api-access-nrdqx\" (UniqueName: \"kubernetes.io/projected/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-kube-api-access-nrdqx\") pod \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\" (UID: \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\") "
Oct 31 07:45:18.142996 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.142966    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8965a35f-4b30-4e02-b22f-62634ae83b31-maas-api-service-ca" pod "8965a35f-4b30-4e02-b22f-62634ae83b31" (UID: "8965a35f-4b30-4e02-b22f-62634ae83b31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:18.143076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.143026    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-openshift-ca" pod "8d035bbe-d467-4f01-8325-8ccac7e65551" (UID: "8d035bbe-d467-4f01-8325-8ccac7e65551"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:18.143488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.143461    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c0428e-b966-4e6e-8425-15bc33a040b6-plugins-config-volume" pod "91c0428e-b966-4e6e-8425-15bc33a040b6" (UID: "91c0428e-b966-4e6e-8425-15bc33a040b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:18.143599 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.143487    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.143599 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.143510    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8965a35f-4b30-4e02-b22f-62634ae83b31-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.143599 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.143527    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.143599 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.143542    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khjh8\" (UniqueName: \"kubernetes.io/projected/8f6c6f59-6834-4d62-9e21-1f263fff5b7b-kube-api-access-khjh8\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.145421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.145386    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8965a35f-4b30-4e02-b22f-62634ae83b31-kube-api-access-ljmzf" pod "8965a35f-4b30-4e02-b22f-62634ae83b31" (UID: "8965a35f-4b30-4e02-b22f-62634ae83b31"). InnerVolumeSpecName "kube-api-access-ljmzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:18.145514 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.145488    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-kube-api-access-nrdqx" pod "d35962de-8dfc-477e-aebe-c0fb2ec68e5e" (UID: "d35962de-8dfc-477e-aebe-c0fb2ec68e5e"). InnerVolumeSpecName "kube-api-access-nrdqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:18.145588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.145540    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d035bbe-d467-4f01-8325-8ccac7e65551-kube-api-access-p6sbl" pod "8d035bbe-d467-4f01-8325-8ccac7e65551" (UID: "8d035bbe-d467-4f01-8325-8ccac7e65551"). InnerVolumeSpecName "kube-api-access-p6sbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:18.146043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.146025    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c0428e-b966-4e6e-8425-15bc33a040b6-kube-api-access-ddlnl" pod "91c0428e-b966-4e6e-8425-15bc33a040b6" (UID: "91c0428e-b966-4e6e-8425-15bc33a040b6"). InnerVolumeSpecName "kube-api-access-ddlnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:18.244005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.243972    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-xks-ca\") pod \"8d035bbe-d467-4f01-8325-8ccac7e65551\" (UID: \"8d035bbe-d467-4f01-8325-8ccac7e65551\") "
Oct 31 07:45:18.244235 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.244014    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-maas-api-service-ca\") pod \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\" (UID: \"d35962de-8dfc-477e-aebe-c0fb2ec68e5e\") "
Oct 31 07:45:18.244552 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.244518    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-maas-api-service-ca" pod "d35962de-8dfc-477e-aebe-c0fb2ec68e5e" (UID: "d35962de-8dfc-477e-aebe-c0fb2ec68e5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:18.244661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.244596    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91c0428e-b966-4e6e-8425-15bc33a040b6-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.244661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.244618    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrdqx\" (UniqueName: \"kubernetes.io/projected/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-kube-api-access-nrdqx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.244661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.244632    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6sbl\" (UniqueName: \"kubernetes.io/projected/8d035bbe-d467-4f01-8325-8ccac7e65551-kube-api-access-p6sbl\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.244661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.244645    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljmzf\" (UniqueName: \"kubernetes.io/projected/8965a35f-4b30-4e02-b22f-62634ae83b31-kube-api-access-ljmzf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.244661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.244657    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddlnl\" (UniqueName: \"kubernetes.io/projected/91c0428e-b966-4e6e-8425-15bc33a040b6-kube-api-access-ddlnl\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.246371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.246270    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-xks-ca" pod "8d035bbe-d467-4f01-8325-8ccac7e65551" (UID: "8d035bbe-d467-4f01-8325-8ccac7e65551"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:18.345862 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.345823    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d035bbe-d467-4f01-8325-8ccac7e65551-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.345862 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.345853    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d35962de-8dfc-477e-aebe-c0fb2ec68e5e-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:18.796496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.796386    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"
Oct 31 07:45:18.796496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.796425    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726" event={"ID":"8f6c6f59-6834-4d62-9e21-1f263fff5b7b","Type":"ContainerDied","Data":"767f41c556fe032d5c5c0ab58824b473201081905f075259407926b00a1508b6"}
Oct 31 07:45:18.797126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.796502    5829 scope.go:122] "RemoveContainer" containerID="f78ba7606153235b5125ad56dea27f82241bc7bb195332807e8048cf797cc5b1"
Oct 31 07:45:18.799091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.799070    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"
Oct 31 07:45:18.799355 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.799078    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f" event={"ID":"91c0428e-b966-4e6e-8425-15bc33a040b6","Type":"ContainerDied","Data":"f572c4253fc4a3d946cb95b487e3daf7ae5589ef9790064f74f8820932bb5cc2"}
Oct 31 07:45:18.802739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.802706    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv" event={"ID":"d35962de-8dfc-477e-aebe-c0fb2ec68e5e","Type":"ContainerDied","Data":"30857cbfec64b08d7b70f34c6b27a61bc946e96414ee173ea391031536e2e187"}
Oct 31 07:45:18.802852 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.802806    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"
Oct 31 07:45:18.808882 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.808859    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl" event={"ID":"8965a35f-4b30-4e02-b22f-62634ae83b31","Type":"ContainerDied","Data":"8b038277282034fdfe24e09e53a78dd25bcd36ce143919e3dab2c17ddceab4bb"}
Oct 31 07:45:18.808974 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.808942    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"
Oct 31 07:45:18.813479 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.813405    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49" event={"ID":"8d035bbe-d467-4f01-8325-8ccac7e65551","Type":"ContainerDied","Data":"3e26a203da1a10cd2e87fee37f0fc95bba24f5157881e7a4b93608a97c99a9ab"}
Oct 31 07:45:18.813479 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.813453    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e26a203da1a10cd2e87fee37f0fc95bba24f5157881e7a4b93608a97c99a9ab"
Oct 31 07:45:18.818391 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.817930    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"
Oct 31 07:45:18.818391 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.818361    5829 scope.go:122] "RemoveContainer" containerID="179a509c109305b296d36c07e45dd5fe48277e0ff7775fba63ff7d076bad7f47"
Oct 31 07:45:18.836068 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.836043    5829 scope.go:122] "RemoveContainer" containerID="bb3f3a556b561de0b9a54988c9ca907ddae4b802c9d41a06153a309d92b665d4"
Oct 31 07:45:18.849338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.849310    5829 scope.go:122] "RemoveContainer" containerID="ef59c1eeaa736d29752ccc1273a9d233019f846584646d901fc6186d6c130001"
Oct 31 07:45:18.952499 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.952185    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"]
Oct 31 07:45:18.953043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.953020    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerName="cleanup"
Oct 31 07:45:18.953154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.953044    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerName="cleanup"
Oct 31 07:45:18.953154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.953068    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerName="cleanup"
Oct 31 07:45:18.953154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.953075    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerName="cleanup"
Oct 31 07:45:18.953154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.953086    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerName="cleanup"
Oct 31 07:45:18.953154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.953094    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerName="cleanup"
Oct 31 07:45:18.953154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.953105    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8d035bbe-d467-4f01-8325-8ccac7e65551" containerName="revoke-keys"
Oct 31 07:45:18.953154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.953112    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d035bbe-d467-4f01-8325-8ccac7e65551" containerName="revoke-keys"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954296    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerName="cleanup"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954320    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerName="cleanup"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954344    5829 cpu_manager.go:414] "containerMap: removing container" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954351    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954375    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8f6c6f59-6834-4d62-9e21-1f263fff5b7b" containerName="payload-processing"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954385    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f6c6f59-6834-4d62-9e21-1f263fff5b7b" containerName="payload-processing"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954396    5829 cpu_manager.go:414] "containerMap: removing container" podUID="91c0428e-b966-4e6e-8425-15bc33a040b6" containerName="payload-pre-processing"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954404    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c0428e-b966-4e6e-8425-15bc33a040b6" containerName="payload-pre-processing"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954420    5829 cpu_manager.go:414] "containerMap: removing container" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954427    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954440    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy"
Oct 31 07:45:18.954496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954447    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d130a1c7-6b3f-4168-af6a-451147ed223d" containerName="istio-proxy"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954660    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" containerName="main"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954680    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerName="cleanup"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954690    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerName="cleanup"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954702    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerName="cleanup"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954711    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d035bbe-d467-4f01-8325-8ccac7e65551" containerName="revoke-keys"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954725    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerName="cleanup"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954735    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerName="cleanup"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954752    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="549cac55-429e-46a1-b33d-a619f959a170" containerName="main"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954764    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f6c6f59-6834-4d62-9e21-1f263fff5b7b" containerName="payload-processing"
Oct 31 07:45:18.955152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.954772    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c0428e-b966-4e6e-8425-15bc33a040b6" containerName="payload-pre-processing"
Oct 31 07:45:18.968866 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.968833    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"]
Oct 31 07:45:18.969017 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.968888    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:18.977438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:18.977409    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-b1fda4-56fbf98588-px726"]
Oct 31 07:45:19.002954 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.002917    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"]
Oct 31 07:45:19.064248 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.064151    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"]
Oct 31 07:45:19.123187 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.122526    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-b1fda4-29890545-vlfjv"]
Oct 31 07:45:19.135379 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.135340    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"]
Oct 31 07:45:19.138217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.138171    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerName="cleanup"
Oct 31 07:45:19.138390 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.138375    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" containerName="cleanup"
Oct 31 07:45:19.138474 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.138464    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerName="cleanup"
Oct 31 07:45:19.138588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.138577    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerName="cleanup"
Oct 31 07:45:19.138920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.138907    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" containerName="cleanup"
Oct 31 07:45:19.146650 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.146614    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"]
Oct 31 07:45:19.147067 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.146903    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.152659 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.152639    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"]
Oct 31 07:45:19.157248 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.157194    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-b4e8fd-29890545-vsvzl"]
Oct 31 07:45:19.160706 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.160679    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-envoy\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.160827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.160725    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-data\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.160827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.160757    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-token\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.160925 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.160842    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acad66-7d35-49b2-8b03-8f900db5c8df-istiod-ca-cert\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.161244 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.161214    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59gld\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-kube-api-access-59gld\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.161791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.161513    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.161791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.161589    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-certs\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.161791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.161620    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-podinfo\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.161791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.161644    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-credential-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.164346 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.164325    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"]
Oct 31 07:45:19.166023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.165998    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:19.170882 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.170857    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-b1fda4-6db8b6c5f5-9cr9f"]
Oct 31 07:45:19.262790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.262749    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348fc3b3-343d-4137-9821-c551fc16dd14-istio-podinfo\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.262790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.262792    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-certs\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.263004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.262881    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vckw\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-kube-api-access-5vckw\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.263004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.262930    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gld\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-kube-api-access-59gld\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263070 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263019    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-istio-token\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.263070 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263051    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-credential-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.263142 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263091    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.263142 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263123    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263264 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263241    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-certs\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263346 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263331    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-podinfo\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263459 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263361    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348fc3b3-343d-4137-9821-c551fc16dd14-istiod-ca-cert\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.263459 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263391    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-credential-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263527 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263485    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-envoy\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263527 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263507    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-envoy\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.263593 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263568    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263634 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263612    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-certs\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263701    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-data\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263823 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263805    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-token\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263895 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263877    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-data\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.263951 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263902    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-credential-socket\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.263951 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263929    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acad66-7d35-49b2-8b03-8f900db5c8df-istiod-ca-cert\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.264039 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.263955    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-data\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.264623 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.264599    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acad66-7d35-49b2-8b03-8f900db5c8df-istiod-ca-cert\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.267288 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.267232    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-podinfo\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.267581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.267555    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-envoy\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.283946 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.283903    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59gld\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-kube-api-access-59gld\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.288040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.288010    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-token\") pod \"e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") " pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.301574 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.301540    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.374885    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993617de-c2f4-4f68-899b-f4cd1b192429-kube-api-access-wcpjc\" (UniqueName: \"kubernetes.io/projected/993617de-c2f4-4f68-899b-f4cd1b192429-kube-api-access-wcpjc\") pod \"993617de-c2f4-4f68-899b-f4cd1b192429\" (UID: \"993617de-c2f4-4f68-899b-f4cd1b192429\") "
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.374985    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993617de-c2f4-4f68-899b-f4cd1b192429-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993617de-c2f4-4f68-899b-f4cd1b192429-maas-api-service-ca\") pod \"993617de-c2f4-4f68-899b-f4cd1b192429\" (UID: \"993617de-c2f4-4f68-899b-f4cd1b192429\") "
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.375664    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348fc3b3-343d-4137-9821-c551fc16dd14-istio-podinfo\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.375735    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-certs\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.375808    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vckw\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-kube-api-access-5vckw\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.375935    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-istio-token\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.375983    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-credential-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.376040    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.376167    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348fc3b3-343d-4137-9821-c551fc16dd14-istiod-ca-cert\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.376226    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993617de-c2f4-4f68-899b-f4cd1b192429-maas-api-service-ca" pod "993617de-c2f4-4f68-899b-f4cd1b192429" (UID: "993617de-c2f4-4f68-899b-f4cd1b192429"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.376301    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-envoy\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.376380    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-data\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.377436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.376508    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/993617de-c2f4-4f68-899b-f4cd1b192429-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:19.389932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.378554    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-data\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.389932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.381047    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-credential-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.389932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.387654    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-certs\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.389932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.388882    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348fc3b3-343d-4137-9821-c551fc16dd14-istiod-ca-cert\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.389932 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.389083    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-socket\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.407689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.407617    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348fc3b3-343d-4137-9821-c551fc16dd14-istio-podinfo\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.411885 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.410601    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-envoy\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.411885 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.411813    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"]
Oct 31 07:45:19.421870 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.421797    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vckw\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-kube-api-access-5vckw\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.424352 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.424260    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-b1fda4-6ntq5"]
Oct 31 07:45:19.425887 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.425828    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-istio-token\") pod \"e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") " pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.429625 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.429577    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993617de-c2f4-4f68-899b-f4cd1b192429-kube-api-access-wcpjc" pod "993617de-c2f4-4f68-899b-f4cd1b192429" (UID: "993617de-c2f4-4f68-899b-f4cd1b192429"). InnerVolumeSpecName "kube-api-access-wcpjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:19.477445 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.477405    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcpjc\" (UniqueName: \"kubernetes.io/projected/993617de-c2f4-4f68-899b-f4cd1b192429-kube-api-access-wcpjc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:19.478863 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.478835    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:19.647172 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.647128    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"]
Oct 31 07:45:19.647360 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:45:19.647333    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39acad66_7d35_49b2_8b03_8f900db5c8df.slice/crio-d32f41785f9d434655a9f2b0edd6355d58bb946500c99a6b1d2467d8e770d477 WatchSource:0}: Error finding container d32f41785f9d434655a9f2b0edd6355d58bb946500c99a6b1d2467d8e770d477: Status 404 returned error can't find the container with id d32f41785f9d434655a9f2b0edd6355d58bb946500c99a6b1d2467d8e770d477
Oct 31 07:45:19.649937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.649899    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:45:19.650043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.649973    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:45:19.650043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.650001    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:45:19.796540 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:45:19.796505    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348fc3b3_343d_4137_9821_c551fc16dd14.slice/crio-7862ddb79784476f95ca3fc3c4829c03730c0b586c6059165a96d2488c715139 WatchSource:0}: Error finding container 7862ddb79784476f95ca3fc3c4829c03730c0b586c6059165a96d2488c715139: Status 404 returned error can't find the container with id 7862ddb79784476f95ca3fc3c4829c03730c0b586c6059165a96d2488c715139
Oct 31 07:45:19.797341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.797317    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"]
Oct 31 07:45:19.799538 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.799505    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:45:19.799624 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.799581    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:45:19.799694 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.799625    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 07:45:19.834038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.834011    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" event={"ID":"348fc3b3-343d-4137-9821-c551fc16dd14","Type":"ContainerStarted","Data":"7862ddb79784476f95ca3fc3c4829c03730c0b586c6059165a96d2488c715139"}
Oct 31 07:45:19.839373 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.837936    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" event={"ID":"39acad66-7d35-49b2-8b03-8f900db5c8df","Type":"ContainerStarted","Data":"c3cf3efab955a372d73a43a6562c5bb6f76de632f0a1e45fcbc6ae64cd64ded2"}
Oct 31 07:45:19.839373 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.838000    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" event={"ID":"39acad66-7d35-49b2-8b03-8f900db5c8df","Type":"ContainerStarted","Data":"d32f41785f9d434655a9f2b0edd6355d58bb946500c99a6b1d2467d8e770d477"}
Oct 31 07:45:19.846741 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.846711    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd" event={"ID":"993617de-c2f4-4f68-899b-f4cd1b192429","Type":"ContainerDied","Data":"7b7fd5928909dd90d780e37afc8496f10ac295f9e6f833295b082a45e9ec1694"}
Oct 31 07:45:19.846901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.846773    5829 scope.go:122] "RemoveContainer" containerID="34292fdac8b270b13db5377d7aaedd14661648ac96565efec44df9c507a510fb"
Oct 31 07:45:19.846970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.846913    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"
Oct 31 07:45:19.890917 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.889827    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podStartSLOduration=1.8898109920000001 podStartE2EDuration="1.889810992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:19.886639631 +0000 UTC m=+984.025800254" watchObservedRunningTime="2026-10-31 07:45:19.889810992 +0000 UTC m=+984.028971580"
Oct 31 07:45:19.931026 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.930974    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"]
Oct 31 07:45:19.939111 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.939073    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3cc201-29890545-fthgd"]
Oct 31 07:45:19.993226 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.993191    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="549cac55-429e-46a1-b33d-a619f959a170" path="/var/lib/kubelet/pods/549cac55-429e-46a1-b33d-a619f959a170/volumes"
Oct 31 07:45:19.993725 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.993701    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8965a35f-4b30-4e02-b22f-62634ae83b31" path="/var/lib/kubelet/pods/8965a35f-4b30-4e02-b22f-62634ae83b31/volumes"
Oct 31 07:45:19.994272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.994088    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f6c6f59-6834-4d62-9e21-1f263fff5b7b" path="/var/lib/kubelet/pods/8f6c6f59-6834-4d62-9e21-1f263fff5b7b/volumes"
Oct 31 07:45:19.994914 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.994860    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f033de-bdf1-430f-94cc-d1fd0b452c9d" path="/var/lib/kubelet/pods/90f033de-bdf1-430f-94cc-d1fd0b452c9d/volumes"
Oct 31 07:45:19.995321 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.995299    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c0428e-b966-4e6e-8425-15bc33a040b6" path="/var/lib/kubelet/pods/91c0428e-b966-4e6e-8425-15bc33a040b6/volumes"
Oct 31 07:45:19.995667 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.995652    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" path="/var/lib/kubelet/pods/993617de-c2f4-4f68-899b-f4cd1b192429/volumes"
Oct 31 07:45:19.996367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.996348    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d35962de-8dfc-477e-aebe-c0fb2ec68e5e" path="/var/lib/kubelet/pods/d35962de-8dfc-477e-aebe-c0fb2ec68e5e/volumes"
Oct 31 07:45:19.996784 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:19.996769    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5" path="/var/lib/kubelet/pods/e8bd0c5e-5c83-4734-bf2b-ff2efb34d2d5/volumes"
Oct 31 07:45:20.302672 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:20.302583    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:20.304452 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:20.304416    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:20.304576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:20.304467    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:20.864243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:20.864200    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" event={"ID":"348fc3b3-343d-4137-9821-c551fc16dd14","Type":"ContainerStarted","Data":"829dba704a1e6b0758d3be7798cab64d2fc1546d8e2f2b3fedfe1d7cb184106e"}
Oct 31 07:45:20.957321 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:20.952460    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podStartSLOduration=1.9524393610000002 podStartE2EDuration="1.952439361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:20.91680036 +0000 UTC m=+985.055960975" watchObservedRunningTime="2026-10-31 07:45:20.952439361 +0000 UTC m=+985.091599949"
Oct 31 07:45:21.302474 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:21.302436    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:21.302661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:21.302495    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:21.479715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:21.479680    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:21.481545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:21.481516    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 07:45:21.481676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:21.481568    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 07:45:21.867832 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:21.867786    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"]
Oct 31 07:45:22.302569 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.302527    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:22.302749 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.302601    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:22.367341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.367298    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"]
Oct 31 07:45:22.368556 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.368534    5829 cpu_manager.go:414] "containerMap: removing container" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" containerName="cleanup"
Oct 31 07:45:22.368680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.368668    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" containerName="cleanup"
Oct 31 07:45:22.368759 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.368750    5829 cpu_manager.go:414] "containerMap: removing container" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" containerName="cleanup"
Oct 31 07:45:22.368851 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.368842    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" containerName="cleanup"
Oct 31 07:45:22.368938 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.368928    5829 cpu_manager.go:414] "containerMap: removing container" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" containerName="cleanup"
Oct 31 07:45:22.369007 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.368999    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" containerName="cleanup"
Oct 31 07:45:22.369249 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.369238    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" containerName="cleanup"
Oct 31 07:45:22.369355 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.369345    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" containerName="cleanup"
Oct 31 07:45:22.375663 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.375640    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.389149 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.389116    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"]
Oct 31 07:45:22.392088 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.392061    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:45:22.392198 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.392113    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:45:22.392252 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.392213    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:45:22.392252 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.392071    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:45:22.393038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.393010    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pmn6\" (UniqueName: \"kubernetes.io/projected/eb1f2998-f127-446f-8f12-216bc91e88a7-kube-api-access-2pmn6\") pod \"maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.393165 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.393145    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.393266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.393238    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.496558 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.495820    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 07:45:22.496558 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.495877    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 07:45:22.497911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.497864    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pmn6\" (UniqueName: \"kubernetes.io/projected/eb1f2998-f127-446f-8f12-216bc91e88a7-kube-api-access-2pmn6\") pod \"maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.498624 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.498576    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.500149 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.500097    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.500425 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.499263    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.535350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.518776    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.569370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.567029    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pmn6\" (UniqueName: \"kubernetes.io/projected/eb1f2998-f127-446f-8f12-216bc91e88a7-kube-api-access-2pmn6\") pod \"maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:22.706842 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:22.706803    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:23.172689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.172663    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"]
Oct 31 07:45:23.302029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.301988    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:23.302204 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.302046    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:23.480323 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.480273    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 07:45:23.480495 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.480345    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 07:45:23.574927 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.573108    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:45:23.581776 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.581746    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"]
Oct 31 07:45:23.582647 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.582624    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="993617de-c2f4-4f68-899b-f4cd1b192429" containerName="cleanup"
Oct 31 07:45:23.588094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.588073    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.588682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.588658    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"]
Oct 31 07:45:23.594471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.594441    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"]
Oct 31 07:45:23.600334 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.600315    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.601344 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.601323    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"]
Oct 31 07:45:23.677104 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.677075    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3061197b-7fwnq\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.677268 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.677250    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3687bccb-9rqzs\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.677396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.677352    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3061197b-7fwnq\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.677396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.677383    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3687bccb-9rqzs\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.677505 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.677421    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrjw2\" (UniqueName: \"kubernetes.io/projected/5c52732f-cee9-421a-ad27-422e304db24e-kube-api-access-hrjw2\") pod \"maas-api-revoke-sub-3061197b-7fwnq\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.677505 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.677494    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt7vk\" (UniqueName: \"kubernetes.io/projected/874be886-01ff-4e96-be4f-fbe8c0ae7334-kube-api-access-vt7vk\") pod \"maas-api-revoke-sub-3687bccb-9rqzs\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.778707 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.778602    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3687bccb-9rqzs\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.779871 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.779849    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3061197b-7fwnq\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.780481 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.780460    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3687bccb-9rqzs\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.781447 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.781427    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrjw2\" (UniqueName: \"kubernetes.io/projected/5c52732f-cee9-421a-ad27-422e304db24e-kube-api-access-hrjw2\") pod \"maas-api-revoke-sub-3061197b-7fwnq\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.781961 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.781941    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt7vk\" (UniqueName: \"kubernetes.io/projected/874be886-01ff-4e96-be4f-fbe8c0ae7334-kube-api-access-vt7vk\") pod \"maas-api-revoke-sub-3687bccb-9rqzs\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.782439 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.781299    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3687bccb-9rqzs\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.783397 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.783161    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3061197b-7fwnq\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.783397 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.783356    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3061197b-7fwnq\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.783606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.783587    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3061197b-7fwnq\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.783874 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.783855    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3687bccb-9rqzs\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.801770 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.801739    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt7vk\" (UniqueName: \"kubernetes.io/projected/874be886-01ff-4e96-be4f-fbe8c0ae7334-kube-api-access-vt7vk\") pod \"maas-api-revoke-sub-3687bccb-9rqzs\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.801878 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.801858    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrjw2\" (UniqueName: \"kubernetes.io/projected/5c52732f-cee9-421a-ad27-422e304db24e-kube-api-access-hrjw2\") pod \"maas-api-revoke-sub-3061197b-7fwnq\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:23.964105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.964064    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2" event={"ID":"eb1f2998-f127-446f-8f12-216bc91e88a7","Type":"ContainerStarted","Data":"7ebbcc2d31c7a0c6de306431ae0c4dba330957ee0b40b623df7de28d8612a3d5"}
Oct 31 07:45:23.964944 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.964609    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2" event={"ID":"eb1f2998-f127-446f-8f12-216bc91e88a7","Type":"ContainerStarted","Data":"d12a0449d26b461c25abaeac42f7853ccd43775531747d57199079f6e760852b"}
Oct 31 07:45:23.980372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.976712    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:23.980372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:23.979027    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:24.031352 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:24.024817    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2" podStartSLOduration=2.024794686 podStartE2EDuration="2.024794686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:24.022988308 +0000 UTC m=+988.162148920" watchObservedRunningTime="2026-10-31 07:45:24.024794686 +0000 UTC m=+988.163955276"
Oct 31 07:45:24.446332 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:24.443592    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:24.446332 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:24.443733    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:24.754335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:24.573386    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 07:45:24.754335 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:24.752037    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 07:45:25.320467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.320411    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:25.327520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.327483    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:25.327772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.327742    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:25.453354 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.445920    5829 generic.go:356] "Generic (PLEG): container finished" podID="eb1f2998-f127-446f-8f12-216bc91e88a7" containerID="7ebbcc2d31c7a0c6de306431ae0c4dba330957ee0b40b623df7de28d8612a3d5" exitCode=0
Oct 31 07:45:25.453354 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.446269    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2" event={"ID":"eb1f2998-f127-446f-8f12-216bc91e88a7","Type":"ContainerDied","Data":"7ebbcc2d31c7a0c6de306431ae0c4dba330957ee0b40b623df7de28d8612a3d5"}
Oct 31 07:45:25.545359 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.523484    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:25.545359 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.531731    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"]
Oct 31 07:45:25.573577 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.573475    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:25.574747 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.574721    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 07:45:25.574911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:25.574891    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 07:45:26.346365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.342454    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:26.346365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.342527    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:26.368306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.348465    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:26.392342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.368574    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:26.472308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.418601    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"]
Oct 31 07:45:26.476307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.472991    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:45:26.487959 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.487920    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 07:45:26.488237 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.488205    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 07:45:26.536621 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.536577    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 07:45:26.537658 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.537630    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:45:26.538646 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.538618    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 31 07:45:26.780308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.595393    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": context deadline exceeded"
Oct 31 07:45:26.781859 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.781824    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:45:26.877376 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.876606    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq" podUID="b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" containerName="payload-processing" containerID="cri-o://ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750" gracePeriod=25
Oct 31 07:45:26.911563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.911520    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs" event={"ID":"874be886-01ff-4e96-be4f-fbe8c0ae7334","Type":"ContainerStarted","Data":"d709b1f44471d09dfd71e765ea1913e44484714630b9248f2bde26cf58982880"}
Oct 31 07:45:26.962961 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:26.962920    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq" event={"ID":"5c52732f-cee9-421a-ad27-422e304db24e","Type":"ContainerStarted","Data":"e2ee6a266b4e444d3baece483b880186a61c68d345d55994eb60f4375ac46350"}
Oct 31 07:45:27.477308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.404360    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 31 07:45:27.477308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.436644    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:27.477308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.437617    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:45:27.477308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.457589    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:27.477308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.457643    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:27.526307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.492629    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 07:45:27.526307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.492688    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 07:45:27.545341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.539390    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:27.763873 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.763823    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:27.764526 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.764500    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:45:27.857308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.795261    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:28.142096 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.962490    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:45:28.146923 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:28.146874    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:28.147166 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.962568    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:45:28.147337 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:28.147312    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:28.152303 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.962625    5829 patch_prober.go:29] interesting pod/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:45:28.152492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:28.152465    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:28.152614 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:27.991322    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr" podUID="df54b1a3-b0e3-4481-b483-250b1136c78d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:28.192173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:28.192139    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:45:28.192756 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:28.192630    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:28.555310 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:28.554462    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4" podUID="0b5c0eeb-8829-4fa6-8ce0-617cbf88b831" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Oct 31 07:45:28.614322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:28.584604    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 31 07:45:29.002849 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.001805    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:29.004900 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.004293    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 07:45:29.006334 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.004356    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 07:45:29.078396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.074221    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 07:45:29.085234 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:29.085174    5829 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.119s"
Oct 31 07:45:29.085434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.085349    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"]
Oct 31 07:45:29.087002 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.086338    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy" containerID="cri-o://7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff" gracePeriod=30
Oct 31 07:45:29.094128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.094096    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:29.155270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.154800    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:29.155270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.147473    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:29.155270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.104395    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"]
Oct 31 07:45:29.155270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.155057    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs" event={"ID":"874be886-01ff-4e96-be4f-fbe8c0ae7334","Type":"ContainerStarted","Data":"19c1dcf8e907286aa73372c942945c315f7de215d36f8c9dcd3d63f97e4a0c63"}
Oct 31 07:45:29.155270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.155151    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:45:29.155270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.155213    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:45:29.155270 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.155270    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:45:29.156037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.155980    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" containerID="cri-o://dc707bdc4b5ae066eb5f7799d82aeecce2f909ccb238d61b37f29c60e5351fb3" gracePeriod=30
Oct 31 07:45:29.171127 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.169900    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:45:29.171127 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.170165    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4"
Oct 31 07:45:29.175498 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.175469    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"]
Oct 31 07:45:29.175955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.175925    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy" containerID="cri-o://c2dd6256bcd4746fe4b1311ce9a4dd3b5197803a73e2b194f426eaa93c01da8d" gracePeriod=30
Oct 31 07:45:29.221485 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.221363    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs" podStartSLOduration=6.221341755 podStartE2EDuration="6.221341755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:29.208406152 +0000 UTC m=+993.347566765" watchObservedRunningTime="2026-10-31 07:45:29.221341755 +0000 UTC m=+993.360502344"
Oct 31 07:45:29.269866 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.269822    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1f2998-f127-446f-8f12-216bc91e88a7-kube-api-access-2pmn6\" (UniqueName: \"kubernetes.io/projected/eb1f2998-f127-446f-8f12-216bc91e88a7-kube-api-access-2pmn6\") pod \"eb1f2998-f127-446f-8f12-216bc91e88a7\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") "
Oct 31 07:45:29.270128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.270106    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-xks-ca\") pod \"eb1f2998-f127-446f-8f12-216bc91e88a7\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") "
Oct 31 07:45:29.270259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.270243    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-openshift-ca\") pod \"eb1f2998-f127-446f-8f12-216bc91e88a7\" (UID: \"eb1f2998-f127-446f-8f12-216bc91e88a7\") "
Oct 31 07:45:29.275985 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.275939    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-openshift-ca" pod "eb1f2998-f127-446f-8f12-216bc91e88a7" (UID: "eb1f2998-f127-446f-8f12-216bc91e88a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:29.280881 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.280850    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1f2998-f127-446f-8f12-216bc91e88a7-kube-api-access-2pmn6" pod "eb1f2998-f127-446f-8f12-216bc91e88a7" (UID: "eb1f2998-f127-446f-8f12-216bc91e88a7"). InnerVolumeSpecName "kube-api-access-2pmn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:29.286728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.286468    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-xks-ca" pod "eb1f2998-f127-446f-8f12-216bc91e88a7" (UID: "eb1f2998-f127-446f-8f12-216bc91e88a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:29.303869 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.303832    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 31 07:45:29.304083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.303890    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 31 07:45:29.378091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.378058    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pmn6\" (UniqueName: \"kubernetes.io/projected/eb1f2998-f127-446f-8f12-216bc91e88a7-kube-api-access-2pmn6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:29.378329 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.378312    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:29.378441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.378429    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1f2998-f127-446f-8f12-216bc91e88a7-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:29.480584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.479863    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body=
Oct 31 07:45:29.480584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:29.479923    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused"
Oct 31 07:45:30.130605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.130424    5829 generic.go:356] "Generic (PLEG): container finished" podID="5c52732f-cee9-421a-ad27-422e304db24e" containerID="e64c29233643d6caf2c8c56d216380fdfe31bc72528ad0dad1482f7c3b0403b0" exitCode=0
Oct 31 07:45:30.130605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.130519    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq" event={"ID":"5c52732f-cee9-421a-ad27-422e304db24e","Type":"ContainerDied","Data":"e64c29233643d6caf2c8c56d216380fdfe31bc72528ad0dad1482f7c3b0403b0"}
Oct 31 07:45:30.132660 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.132553    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"
Oct 31 07:45:30.132660 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.132584    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2" event={"ID":"eb1f2998-f127-446f-8f12-216bc91e88a7","Type":"ContainerDied","Data":"d12a0449d26b461c25abaeac42f7853ccd43775531747d57199079f6e760852b"}
Oct 31 07:45:30.132660 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.132629    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d12a0449d26b461c25abaeac42f7853ccd43775531747d57199079f6e760852b"
Oct 31 07:45:30.136732 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.136705    5829 generic.go:356] "Generic (PLEG): container finished" podID="874be886-01ff-4e96-be4f-fbe8c0ae7334" containerID="19c1dcf8e907286aa73372c942945c315f7de215d36f8c9dcd3d63f97e4a0c63" exitCode=0
Oct 31 07:45:30.137094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.137049    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs" event={"ID":"874be886-01ff-4e96-be4f-fbe8c0ae7334","Type":"ContainerDied","Data":"19c1dcf8e907286aa73372c942945c315f7de215d36f8c9dcd3d63f97e4a0c63"}
Oct 31 07:45:30.743304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.742810    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:30.743807 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.743788    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:30.744815 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:30.744787    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:45:31.077551 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.077450    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" containerID="cri-o://d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750" gracePeriod=45
Oct 31 07:45:31.086585 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.086547    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" containerID="cri-o://fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43" gracePeriod=45
Oct 31 07:45:31.099272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.099217    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" containerID="cri-o://4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1" gracePeriod=45
Oct 31 07:45:31.364524 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.363869    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused"
Oct 31 07:45:31.528306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.526817    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:45:31.528306 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.526888    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:31.801005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.800970    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:31.834693 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.834651    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:45:31.964365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.963723    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-openshift-ca\") pod \"874be886-01ff-4e96-be4f-fbe8c0ae7334\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") "
Oct 31 07:45:31.964365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.963825    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tls-certs\") pod \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") "
Oct 31 07:45:31.964365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.963928    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-xks-ca\") pod \"874be886-01ff-4e96-be4f-fbe8c0ae7334\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") "
Oct 31 07:45:31.964365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.964013    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tmp-dir\") pod \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") "
Oct 31 07:45:31.964365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.964093    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-kube-api-access-qqbml\" (UniqueName: \"kubernetes.io/projected/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-kube-api-access-qqbml\") pod \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") "
Oct 31 07:45:31.964365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.964145    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-model-cache\") pod \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") "
Oct 31 07:45:31.964365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.964179    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-home\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-home\") pod \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") "
Oct 31 07:45:31.964365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.964224    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874be886-01ff-4e96-be4f-fbe8c0ae7334-kube-api-access-vt7vk\" (UniqueName: \"kubernetes.io/projected/874be886-01ff-4e96-be4f-fbe8c0ae7334-kube-api-access-vt7vk\") pod \"874be886-01ff-4e96-be4f-fbe8c0ae7334\" (UID: \"874be886-01ff-4e96-be4f-fbe8c0ae7334\") "
Oct 31 07:45:31.964365 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:31.964311    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-dshm\") pod \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\" (UID: \"3f55dae4-8d1e-4dba-a2d4-b1e791b39157\") "
Oct 31 07:45:32.011550 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.011496    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-openshift-ca" pod "874be886-01ff-4e96-be4f-fbe8c0ae7334" (UID: "874be886-01ff-4e96-be4f-fbe8c0ae7334"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:32.034040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.033998    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-kube-api-access-qqbml" pod "3f55dae4-8d1e-4dba-a2d4-b1e791b39157" (UID: "3f55dae4-8d1e-4dba-a2d4-b1e791b39157"). InnerVolumeSpecName "kube-api-access-qqbml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:32.039821 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.039777    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tmp-dir" pod "3f55dae4-8d1e-4dba-a2d4-b1e791b39157" (UID: "3f55dae4-8d1e-4dba-a2d4-b1e791b39157"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.075442 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.075391    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-home" pod "3f55dae4-8d1e-4dba-a2d4-b1e791b39157" (UID: "3f55dae4-8d1e-4dba-a2d4-b1e791b39157"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.075938 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.075911    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-model-cache" pod "3f55dae4-8d1e-4dba-a2d4-b1e791b39157" (UID: "3f55dae4-8d1e-4dba-a2d4-b1e791b39157"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.080542 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.080497    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tls-certs" pod "3f55dae4-8d1e-4dba-a2d4-b1e791b39157" (UID: "3f55dae4-8d1e-4dba-a2d4-b1e791b39157"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:32.080542 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.080520    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.080713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.080552    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.080713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.080571    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqbml\" (UniqueName: \"kubernetes.io/projected/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-kube-api-access-qqbml\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.080713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.080583    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.080713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.080596    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.083901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.083872    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-xks-ca" pod "874be886-01ff-4e96-be4f-fbe8c0ae7334" (UID: "874be886-01ff-4e96-be4f-fbe8c0ae7334"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:32.084110 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.084088    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-dshm" pod "3f55dae4-8d1e-4dba-a2d4-b1e791b39157" (UID: "3f55dae4-8d1e-4dba-a2d4-b1e791b39157"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.084924 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.084899    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874be886-01ff-4e96-be4f-fbe8c0ae7334-kube-api-access-vt7vk" pod "874be886-01ff-4e96-be4f-fbe8c0ae7334" (UID: "874be886-01ff-4e96-be4f-fbe8c0ae7334"). InnerVolumeSpecName "kube-api-access-vt7vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:32.091300 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.090613    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:45:32.182542 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.182492    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:32.183984 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.183955    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vt7vk\" (UniqueName: \"kubernetes.io/projected/874be886-01ff-4e96-be4f-fbe8c0ae7334-kube-api-access-vt7vk\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.184172 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.184152    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.184265 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.184253    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f55dae4-8d1e-4dba-a2d4-b1e791b39157-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.184396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.184382    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874be886-01ff-4e96-be4f-fbe8c0ae7334-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.203244 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.203213    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:45:32.232193 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.232155    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs" event={"ID":"874be886-01ff-4e96-be4f-fbe8c0ae7334","Type":"ContainerDied","Data":"d709b1f44471d09dfd71e765ea1913e44484714630b9248f2bde26cf58982880"}
Oct 31 07:45:32.289416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.268417    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d709b1f44471d09dfd71e765ea1913e44484714630b9248f2bde26cf58982880"
Oct 31 07:45:32.289416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.268462    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" event={"ID":"3f55dae4-8d1e-4dba-a2d4-b1e791b39157","Type":"ContainerDied","Data":"fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43"}
Oct 31 07:45:32.289416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.268501    5829 scope.go:122] "RemoveContainer" containerID="fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43"
Oct 31 07:45:32.289416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.246405    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"
Oct 31 07:45:32.289416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.258334    5829 generic.go:356] "Generic (PLEG): container finished" podID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerID="fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43" exitCode=0
Oct 31 07:45:32.289416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.269152    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" event={"ID":"3f55dae4-8d1e-4dba-a2d4-b1e791b39157","Type":"ContainerDied","Data":"ca2667a9339a02b10281dcddda69effb4d5aa82cd0cc55554bb05347e6c99ef2"}
Oct 31 07:45:32.289416 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.258416    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"
Oct 31 07:45:32.298091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.298025    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c52732f-cee9-421a-ad27-422e304db24e-kube-api-access-hrjw2\" (UniqueName: \"kubernetes.io/projected/5c52732f-cee9-421a-ad27-422e304db24e-kube-api-access-hrjw2\") pod \"5c52732f-cee9-421a-ad27-422e304db24e\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") "
Oct 31 07:45:32.298459 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.298345    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-home\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-home\") pod \"b2cac2f4-192d-4553-87fb-a0961517fc68\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") "
Oct 31 07:45:32.298459 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.298417    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-dshm\") pod \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") "
Oct 31 07:45:32.311847 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.311566    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tls-certs\") pod \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") "
Oct 31 07:45:32.312152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.312058    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-dshm\") pod \"b2cac2f4-192d-4553-87fb-a0961517fc68\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") "
Oct 31 07:45:32.312445 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.312388    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2cac2f4-192d-4553-87fb-a0961517fc68-kube-api-access-fchzc\" (UniqueName: \"kubernetes.io/projected/b2cac2f4-192d-4553-87fb-a0961517fc68-kube-api-access-fchzc\") pod \"b2cac2f4-192d-4553-87fb-a0961517fc68\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") "
Oct 31 07:45:32.312599 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.312564    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-model-cache\") pod \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") "
Oct 31 07:45:32.312832 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.312789    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-home\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-home\") pod \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") "
Oct 31 07:45:32.313064 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.313033    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-xks-ca\") pod \"5c52732f-cee9-421a-ad27-422e304db24e\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") "
Oct 31 07:45:32.319126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.314009    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq" event={"ID":"5c52732f-cee9-421a-ad27-422e304db24e","Type":"ContainerDied","Data":"e2ee6a266b4e444d3baece483b880186a61c68d345d55994eb60f4375ac46350"}
Oct 31 07:45:32.319126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.314073    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ee6a266b4e444d3baece483b880186a61c68d345d55994eb60f4375ac46350"
Oct 31 07:45:32.319126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.314166    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"
Oct 31 07:45:32.320702 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.319572    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tmp-dir\") pod \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") "
Oct 31 07:45:32.336616 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.336540    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2cac2f4-192d-4553-87fb-a0961517fc68-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cac2f4-192d-4553-87fb-a0961517fc68-tls-certs\") pod \"b2cac2f4-192d-4553-87fb-a0961517fc68\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") "
Oct 31 07:45:32.338791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.338704    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-openshift-ca\") pod \"5c52732f-cee9-421a-ad27-422e304db24e\" (UID: \"5c52732f-cee9-421a-ad27-422e304db24e\") "
Oct 31 07:45:32.339329 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.339211    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-model-cache\") pod \"b2cac2f4-192d-4553-87fb-a0961517fc68\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") "
Oct 31 07:45:32.339563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.339470    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-kube-api-access-xcg4t\" (UniqueName: \"kubernetes.io/projected/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-kube-api-access-xcg4t\") pod \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\" (UID: \"4b33107c-7771-44a0-8600-4ab5d2b8d1e4\") "
Oct 31 07:45:32.339563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.339535    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-tmp-dir\") pod \"b2cac2f4-192d-4553-87fb-a0961517fc68\" (UID: \"b2cac2f4-192d-4553-87fb-a0961517fc68\") "
Oct 31 07:45:32.341707 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.341682    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-model-cache" pod "b2cac2f4-192d-4553-87fb-a0961517fc68" (UID: "b2cac2f4-192d-4553-87fb-a0961517fc68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.347121 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.343009    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tmp-dir" pod "4b33107c-7771-44a0-8600-4ab5d2b8d1e4" (UID: "4b33107c-7771-44a0-8600-4ab5d2b8d1e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.347370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.343703    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-openshift-ca" pod "5c52732f-cee9-421a-ad27-422e304db24e" (UID: "5c52732f-cee9-421a-ad27-422e304db24e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:32.347505 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.344505    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-tmp-dir" pod "b2cac2f4-192d-4553-87fb-a0961517fc68" (UID: "b2cac2f4-192d-4553-87fb-a0961517fc68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.347628 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.345529    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-model-cache" pod "4b33107c-7771-44a0-8600-4ab5d2b8d1e4" (UID: "4b33107c-7771-44a0-8600-4ab5d2b8d1e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.347739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.345705    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-home" pod "4b33107c-7771-44a0-8600-4ab5d2b8d1e4" (UID: "4b33107c-7771-44a0-8600-4ab5d2b8d1e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.365936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.364024    5829 generic.go:356] "Generic (PLEG): container finished" podID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerID="4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1" exitCode=0
Oct 31 07:45:32.365936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.364212    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" event={"ID":"4b33107c-7771-44a0-8600-4ab5d2b8d1e4","Type":"ContainerDied","Data":"4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1"}
Oct 31 07:45:32.365936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.364252    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt" event={"ID":"4b33107c-7771-44a0-8600-4ab5d2b8d1e4","Type":"ContainerDied","Data":"bc4a87d3d3f4453f5009492952f0f2f7458c7917c119272d42e3433c48be3886"}
Oct 31 07:45:32.365936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.364327    5829 scope.go:122] "RemoveContainer" containerID="fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43"
Oct 31 07:45:32.365936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.364454    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"
Oct 31 07:45:32.365936 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:32.365460    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43\": container with ID starting with fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43 not found: ID does not exist" containerID="fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43"
Oct 31 07:45:32.365936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.365498    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43"} err="failed to get container status \"fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43\": rpc error: code = NotFound desc = could not find container \"fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43\": container with ID starting with fd709d3843239efb1d7b1848f78f966313abc555cd89cd1964e84819f094af43 not found: ID does not exist"
Oct 31 07:45:32.365936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.365519    5829 scope.go:122] "RemoveContainer" containerID="4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1"
Oct 31 07:45:32.378619 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.378586    5829 generic.go:356] "Generic (PLEG): container finished" podID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerID="d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750" exitCode=0
Oct 31 07:45:32.380036 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.379987    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"
Oct 31 07:45:32.380371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.380347    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" event={"ID":"b2cac2f4-192d-4553-87fb-a0961517fc68","Type":"ContainerDied","Data":"d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750"}
Oct 31 07:45:32.380526 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.380508    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" event={"ID":"b2cac2f4-192d-4553-87fb-a0961517fc68","Type":"ContainerDied","Data":"2973159d7d00dfbf2e14f0f9949c9b20270326844d473af07ed3e373cb4d8102"}
Oct 31 07:45:32.385323 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.385273    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-home" pod "b2cac2f4-192d-4553-87fb-a0961517fc68" (UID: "b2cac2f4-192d-4553-87fb-a0961517fc68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.417307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.400225    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tls-certs" pod "4b33107c-7771-44a0-8600-4ab5d2b8d1e4" (UID: "4b33107c-7771-44a0-8600-4ab5d2b8d1e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:32.417307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.403200    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-kube-api-access-xcg4t" pod "4b33107c-7771-44a0-8600-4ab5d2b8d1e4" (UID: "4b33107c-7771-44a0-8600-4ab5d2b8d1e4"). InnerVolumeSpecName "kube-api-access-xcg4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:32.417307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.403874    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c52732f-cee9-421a-ad27-422e304db24e-kube-api-access-hrjw2" pod "5c52732f-cee9-421a-ad27-422e304db24e" (UID: "5c52732f-cee9-421a-ad27-422e304db24e"). InnerVolumeSpecName "kube-api-access-hrjw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:32.417307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.403940    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-xks-ca" pod "5c52732f-cee9-421a-ad27-422e304db24e" (UID: "5c52732f-cee9-421a-ad27-422e304db24e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:32.417307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.404075    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-dshm" pod "4b33107c-7771-44a0-8600-4ab5d2b8d1e4" (UID: "4b33107c-7771-44a0-8600-4ab5d2b8d1e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.417307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.404132    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-dshm" pod "b2cac2f4-192d-4553-87fb-a0961517fc68" (UID: "b2cac2f4-192d-4553-87fb-a0961517fc68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:32.417307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.404415    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2cac2f4-192d-4553-87fb-a0961517fc68-tls-certs" pod "b2cac2f4-192d-4553-87fb-a0961517fc68" (UID: "b2cac2f4-192d-4553-87fb-a0961517fc68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:32.424010 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.423325    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:32.424460 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.424431    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:32.429844 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.429807    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cac2f4-192d-4553-87fb-a0961517fc68-kube-api-access-fchzc" pod "b2cac2f4-192d-4553-87fb-a0961517fc68" (UID: "b2cac2f4-192d-4553-87fb-a0961517fc68"). InnerVolumeSpecName "kube-api-access-fchzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:32.443582 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443541    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443582 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443585    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchzc\" (UniqueName: \"kubernetes.io/projected/b2cac2f4-192d-4553-87fb-a0961517fc68-kube-api-access-fchzc\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443602    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443617    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443632    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443645    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443660    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2cac2f4-192d-4553-87fb-a0961517fc68-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443675    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c52732f-cee9-421a-ad27-422e304db24e-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443689    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443703    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcg4t\" (UniqueName: \"kubernetes.io/projected/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-kube-api-access-xcg4t\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443717    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443732    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrjw2\" (UniqueName: \"kubernetes.io/projected/5c52732f-cee9-421a-ad27-422e304db24e-kube-api-access-hrjw2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443744    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2cac2f4-192d-4553-87fb-a0961517fc68-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443757    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.443830 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.443770    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b33107c-7771-44a0-8600-4ab5d2b8d1e4-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:32.445257 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.445231    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:32.466135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.457186    5829 scope.go:122] "RemoveContainer" containerID="4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1"
Oct 31 07:45:32.466135 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:32.457669    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1\": container with ID starting with 4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1 not found: ID does not exist" containerID="4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1"
Oct 31 07:45:32.466135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.457710    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1"} err="failed to get container status \"4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1\": rpc error: code = NotFound desc = could not find container \"4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1\": container with ID starting with 4b9a7cf6faf65eb3ee9a49de0a8d81e795d0ef860c355ca4e3b8db7393086ce1 not found: ID does not exist"
Oct 31 07:45:32.466135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.457741    5829 scope.go:122] "RemoveContainer" containerID="d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750"
Oct 31 07:45:32.466135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.463597    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"]
Oct 31 07:45:32.472798 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.472722    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9"]
Oct 31 07:45:32.477835 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.477763    5829 scope.go:122] "RemoveContainer" containerID="d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750"
Oct 31 07:45:32.478372 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:32.478310    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750\": container with ID starting with d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750 not found: ID does not exist" containerID="d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750"
Oct 31 07:45:32.478372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.478344    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750"} err="failed to get container status \"d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750\": rpc error: code = NotFound desc = could not find container \"d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750\": container with ID starting with d0ee438acb23d2a4f4ddca734e6044dfa8d598f774fd42ed06f9cf3996413750 not found: ID does not exist"
Oct 31 07:45:32.489902 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.489362    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-w3-3cc201-kserve-84b76c596b-8lpj9" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Oct 31 07:45:32.525007 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.524158    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:45:32.531746 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.531613    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 07:45:32.531746 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.531667    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:45:32.726179 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.726142    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"]
Oct 31 07:45:32.734169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.734135    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/premium-w3-3cc201-kserve-669c9cd68c-wmhxt"]
Oct 31 07:45:32.769578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.769365    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"]
Oct 31 07:45:32.789034 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.789000    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3cc201/distinct-2-w3-3cc201-kserve-5c56557777-66lw2"]
Oct 31 07:45:32.857636 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.857599    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"]
Oct 31 07:45:32.915896 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:32.915862    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"]
Oct 31 07:45:33.303404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.294623    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:45:33.345199 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.345157    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df54b1a3-b0e3-4481-b483-250b1136c78d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df54b1a3-b0e3-4481-b483-250b1136c78d-plugins-config-volume\") pod \"df54b1a3-b0e3-4481-b483-250b1136c78d\" (UID: \"df54b1a3-b0e3-4481-b483-250b1136c78d\") "
Oct 31 07:45:33.345439 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.345255    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df54b1a3-b0e3-4481-b483-250b1136c78d-kube-api-access-ktb65\" (UniqueName: \"kubernetes.io/projected/df54b1a3-b0e3-4481-b483-250b1136c78d-kube-api-access-ktb65\") pod \"df54b1a3-b0e3-4481-b483-250b1136c78d\" (UID: \"df54b1a3-b0e3-4481-b483-250b1136c78d\") "
Oct 31 07:45:33.346887 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.346846    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df54b1a3-b0e3-4481-b483-250b1136c78d-plugins-config-volume" pod "df54b1a3-b0e3-4481-b483-250b1136c78d" (UID: "df54b1a3-b0e3-4481-b483-250b1136c78d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:33.348559 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.348530    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df54b1a3-b0e3-4481-b483-250b1136c78d-kube-api-access-ktb65" pod "df54b1a3-b0e3-4481-b483-250b1136c78d" (UID: "df54b1a3-b0e3-4481-b483-250b1136c78d"). InnerVolumeSpecName "kube-api-access-ktb65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:33.455902 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.455863    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df54b1a3-b0e3-4481-b483-250b1136c78d-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:33.456478 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.456452    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktb65\" (UniqueName: \"kubernetes.io/projected/df54b1a3-b0e3-4481-b483-250b1136c78d-kube-api-access-ktb65\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:33.457533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.457114    5829 patch_prober.go:29] interesting pod/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 07:45:33.457533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.457185    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:45:33.509800 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.509744    5829 generic.go:356] "Generic (PLEG): container finished" podID="df54b1a3-b0e3-4481-b483-250b1136c78d" containerID="b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3" exitCode=0
Oct 31 07:45:33.510534 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.510290    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"
Oct 31 07:45:33.512507 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.510315    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr" event={"ID":"df54b1a3-b0e3-4481-b483-250b1136c78d","Type":"ContainerDied","Data":"b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3"}
Oct 31 07:45:33.512507 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.511634    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr" event={"ID":"df54b1a3-b0e3-4481-b483-250b1136c78d","Type":"ContainerDied","Data":"24c5347a98fee057ffe48f35bd5594ca40c0ed5840321836a5a2bf72668d9990"}
Oct 31 07:45:33.512507 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.511675    5829 scope.go:122] "RemoveContainer" containerID="b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3"
Oct 31 07:45:33.531411 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.530538    5829 scope.go:122] "RemoveContainer" containerID="b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3"
Oct 31 07:45:33.531411 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:33.530948    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3\": container with ID starting with b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3 not found: ID does not exist" containerID="b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3"
Oct 31 07:45:33.531411 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.531018    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3"} err="failed to get container status \"b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3\": rpc error: code = NotFound desc = could not find container \"b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3\": container with ID starting with b6a694efb25c02cd1b20a3e305b69f2f6a4fbae8b1a2d27103b50a6d8b6845d3 not found: ID does not exist"
Oct 31 07:45:33.584524 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.584422    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"]
Oct 31 07:45:33.589599 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.589563    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3dff3f-6448768566-trvmr"]
Oct 31 07:45:33.982127 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.982084    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" path="/var/lib/kubelet/pods/3f55dae4-8d1e-4dba-a2d4-b1e791b39157/volumes"
Oct 31 07:45:33.982609 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.982588    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" path="/var/lib/kubelet/pods/4b33107c-7771-44a0-8600-4ab5d2b8d1e4/volumes"
Oct 31 07:45:33.983010 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.982993    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" path="/var/lib/kubelet/pods/b2cac2f4-192d-4553-87fb-a0961517fc68/volumes"
Oct 31 07:45:33.983809 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:33.983784    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df54b1a3-b0e3-4481-b483-250b1136c78d" path="/var/lib/kubelet/pods/df54b1a3-b0e3-4481-b483-250b1136c78d/volumes"
Oct 31 07:45:34.439753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.438867    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:45:34.558449 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.558357    5829 generic.go:356] "Generic (PLEG): container finished" podID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerID="c2dd6256bcd4746fe4b1311ce9a4dd3b5197803a73e2b194f426eaa93c01da8d" exitCode=0
Oct 31 07:45:34.558994 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.558964    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" event={"ID":"ef061dbf-d524-4e2b-a234-f14abfd09111","Type":"ContainerDied","Data":"c2dd6256bcd4746fe4b1311ce9a4dd3b5197803a73e2b194f426eaa93c01da8d"}
Oct 31 07:45:34.559099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.559037    5829 scope.go:122] "RemoveContainer" containerID="1b806d8aaf8d2d23f88775832188ac68d5fc3995012195c09a3f010502f1f8d7"
Oct 31 07:45:34.570509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.566582    5829 generic.go:356] "Generic (PLEG): container finished" podID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerID="7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff" exitCode=0
Oct 31 07:45:34.570509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.566774    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" event={"ID":"05a46c7a-6f14-42d9-9012-f4177e2fbad8","Type":"ContainerDied","Data":"7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff"}
Oct 31 07:45:34.570509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.566825    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c" event={"ID":"05a46c7a-6f14-42d9-9012-f4177e2fbad8","Type":"ContainerDied","Data":"efd43133e3ff5551c8068e49f9e604380f19d3ae4b684ad576a61acc400119f1"}
Oct 31 07:45:34.570509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.566904    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"
Oct 31 07:45:34.570509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.569942    5829 generic.go:356] "Generic (PLEG): container finished" podID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerID="dc707bdc4b5ae066eb5f7799d82aeecce2f909ccb238d61b37f29c60e5351fb3" exitCode=0
Oct 31 07:45:34.571110 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.571080    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" event={"ID":"7e57edca-955c-4caa-ba2f-4c58e3be16a4","Type":"ContainerDied","Data":"dc707bdc4b5ae066eb5f7799d82aeecce2f909ccb238d61b37f29c60e5351fb3"}
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.601114    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-token\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-token\") pod \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") "
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.601385    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-certs\") pod \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") "
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.601434    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-kube-api-access-b5752\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-kube-api-access-b5752\") pod \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") "
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.601581    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-data\") pod \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") "
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.601738    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-envoy\") pod \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") "
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.601994    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-podinfo\") pod \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") "
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.602172    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-socket\") pod \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") "
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.602338    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istiod-ca-cert\") pod \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") "
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.602612    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-credential-socket\") pod \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\" (UID: \"05a46c7a-6f14-42d9-9012-f4177e2fbad8\") "
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.604067    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-socket" pod "05a46c7a-6f14-42d9-9012-f4177e2fbad8" (UID: "05a46c7a-6f14-42d9-9012-f4177e2fbad8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.604362    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-credential-socket" pod "05a46c7a-6f14-42d9-9012-f4177e2fbad8" (UID: "05a46c7a-6f14-42d9-9012-f4177e2fbad8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.604469    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istiod-ca-cert" pod "05a46c7a-6f14-42d9-9012-f4177e2fbad8" (UID: "05a46c7a-6f14-42d9-9012-f4177e2fbad8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.604580    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-certs" pod "05a46c7a-6f14-42d9-9012-f4177e2fbad8" (UID: "05a46c7a-6f14-42d9-9012-f4177e2fbad8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.610673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.610234    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-data" pod "05a46c7a-6f14-42d9-9012-f4177e2fbad8" (UID: "05a46c7a-6f14-42d9-9012-f4177e2fbad8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.635053 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.612102    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-envoy" pod "05a46c7a-6f14-42d9-9012-f4177e2fbad8" (UID: "05a46c7a-6f14-42d9-9012-f4177e2fbad8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.635053 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.624390    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-podinfo" pod "05a46c7a-6f14-42d9-9012-f4177e2fbad8" (UID: "05a46c7a-6f14-42d9-9012-f4177e2fbad8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:34.635053 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.624628    5829 scope.go:122] "RemoveContainer" containerID="7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff"
Oct 31 07:45:34.635053 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.624884    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-kube-api-access-b5752" pod "05a46c7a-6f14-42d9-9012-f4177e2fbad8" (UID: "05a46c7a-6f14-42d9-9012-f4177e2fbad8"). InnerVolumeSpecName "kube-api-access-b5752". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:34.635053 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.627336    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-token" pod "05a46c7a-6f14-42d9-9012-f4177e2fbad8" (UID: "05a46c7a-6f14-42d9-9012-f4177e2fbad8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:34.645115 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.643392    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:45:34.647326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.646840    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:45:34.660759 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.660635    5829 scope.go:122] "RemoveContainer" containerID="7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff"
Oct 31 07:45:34.667029 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:34.666874    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff\": container with ID starting with 7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff not found: ID does not exist" containerID="7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff"
Oct 31 07:45:34.667029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.666920    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff"} err="failed to get container status \"7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff\": rpc error: code = NotFound desc = could not find container \"7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff\": container with ID starting with 7f88bce5461be56f3bed92cb778b88c2435563c0fe0747470ef97b4aa3796fff not found: ID does not exist"
Oct 31 07:45:34.709326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.706016    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.709326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.706163    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.709326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.706192    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5752\" (UniqueName: \"kubernetes.io/projected/05a46c7a-6f14-42d9-9012-f4177e2fbad8-kube-api-access-b5752\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.709326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.706207    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.709326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.706223    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.709326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.706236    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.709326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.706250    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.709326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.706262    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05a46c7a-6f14-42d9-9012-f4177e2fbad8-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.709326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.706275    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05a46c7a-6f14-42d9-9012-f4177e2fbad8-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.808388 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.808302    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-data\") pod \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") "
Oct 31 07:45:34.808924 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.808840    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-data" pod "7e57edca-955c-4caa-ba2f-4c58e3be16a4" (UID: "7e57edca-955c-4caa-ba2f-4c58e3be16a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.809166 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809131    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-podinfo\") pod \"ef061dbf-d524-4e2b-a234-f14abfd09111\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") "
Oct 31 07:45:34.809275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809240    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-envoy\") pod \"ef061dbf-d524-4e2b-a234-f14abfd09111\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") "
Oct 31 07:45:34.809275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809266    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-token\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-token\") pod \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") "
Oct 31 07:45:34.809401 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809382    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-certs\") pod \"ef061dbf-d524-4e2b-a234-f14abfd09111\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") "
Oct 31 07:45:34.809467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809432    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-certs\") pod \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") "
Oct 31 07:45:34.809519 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809464    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-credential-socket\") pod \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") "
Oct 31 07:45:34.809519 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809500    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-podinfo\") pod \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") "
Oct 31 07:45:34.809796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809770    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-certs" pod "7e57edca-955c-4caa-ba2f-4c58e3be16a4" (UID: "7e57edca-955c-4caa-ba2f-4c58e3be16a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.809796 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809756    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-certs" pod "ef061dbf-d524-4e2b-a234-f14abfd09111" (UID: "ef061dbf-d524-4e2b-a234-f14abfd09111"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.809944 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809822    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-credential-socket" pod "7e57edca-955c-4caa-ba2f-4c58e3be16a4" (UID: "7e57edca-955c-4caa-ba2f-4c58e3be16a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.810003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.809983    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-credential-socket\") pod \"ef061dbf-d524-4e2b-a234-f14abfd09111\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") "
Oct 31 07:45:34.810314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.810254    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-credential-socket" pod "ef061dbf-d524-4e2b-a234-f14abfd09111" (UID: "ef061dbf-d524-4e2b-a234-f14abfd09111"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.810473 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.810454    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef061dbf-d524-4e2b-a234-f14abfd09111-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef061dbf-d524-4e2b-a234-f14abfd09111-istiod-ca-cert\") pod \"ef061dbf-d524-4e2b-a234-f14abfd09111\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") "
Oct 31 07:45:34.810621 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.810594    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istiod-ca-cert\") pod \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") "
Oct 31 07:45:34.810777 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.810753    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-envoy\") pod \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") "
Oct 31 07:45:34.810878 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.810836    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-token\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-token\") pod \"ef061dbf-d524-4e2b-a234-f14abfd09111\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") "
Oct 31 07:45:34.810878 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.810870    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-data\") pod \"ef061dbf-d524-4e2b-a234-f14abfd09111\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") "
Oct 31 07:45:34.810977 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.810958    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-kube-api-access-x49d6\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-kube-api-access-x49d6\") pod \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") "
Oct 31 07:45:34.811024 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.810999    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-socket\") pod \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\" (UID: \"7e57edca-955c-4caa-ba2f-4c58e3be16a4\") "
Oct 31 07:45:34.811125 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.811109    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-kube-api-access-26lzk\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-kube-api-access-26lzk\") pod \"ef061dbf-d524-4e2b-a234-f14abfd09111\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") "
Oct 31 07:45:34.811181 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.811147    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-socket\") pod \"ef061dbf-d524-4e2b-a234-f14abfd09111\" (UID: \"ef061dbf-d524-4e2b-a234-f14abfd09111\") "
Oct 31 07:45:34.811902 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.811880    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.812036 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.812022    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.812135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.812124    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.812223 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.812211    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.812396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.812381    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.812714 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.812275    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-podinfo" pod "ef061dbf-d524-4e2b-a234-f14abfd09111" (UID: "ef061dbf-d524-4e2b-a234-f14abfd09111"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:34.812810 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.812732    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-socket" pod "ef061dbf-d524-4e2b-a234-f14abfd09111" (UID: "ef061dbf-d524-4e2b-a234-f14abfd09111"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.812965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.812947    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-socket" pod "7e57edca-955c-4caa-ba2f-4c58e3be16a4" (UID: "7e57edca-955c-4caa-ba2f-4c58e3be16a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.813006 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.812957    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istiod-ca-cert" pod "7e57edca-955c-4caa-ba2f-4c58e3be16a4" (UID: "7e57edca-955c-4caa-ba2f-4c58e3be16a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:34.813054 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.813032    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef061dbf-d524-4e2b-a234-f14abfd09111-istiod-ca-cert" pod "ef061dbf-d524-4e2b-a234-f14abfd09111" (UID: "ef061dbf-d524-4e2b-a234-f14abfd09111"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:34.813092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.813066    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-token" pod "7e57edca-955c-4caa-ba2f-4c58e3be16a4" (UID: "7e57edca-955c-4caa-ba2f-4c58e3be16a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:34.813332 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.813310    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-data" pod "ef061dbf-d524-4e2b-a234-f14abfd09111" (UID: "ef061dbf-d524-4e2b-a234-f14abfd09111"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.813421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.813377    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-envoy" pod "ef061dbf-d524-4e2b-a234-f14abfd09111" (UID: "ef061dbf-d524-4e2b-a234-f14abfd09111"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.813701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.813679    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-token" pod "ef061dbf-d524-4e2b-a234-f14abfd09111" (UID: "ef061dbf-d524-4e2b-a234-f14abfd09111"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:34.814542 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.814520    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-kube-api-access-x49d6" pod "7e57edca-955c-4caa-ba2f-4c58e3be16a4" (UID: "7e57edca-955c-4caa-ba2f-4c58e3be16a4"). InnerVolumeSpecName "kube-api-access-x49d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:34.814928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.814907    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-kube-api-access-26lzk" pod "ef061dbf-d524-4e2b-a234-f14abfd09111" (UID: "ef061dbf-d524-4e2b-a234-f14abfd09111"). InnerVolumeSpecName "kube-api-access-26lzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:34.815378 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.815353    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-envoy" pod "7e57edca-955c-4caa-ba2f-4c58e3be16a4" (UID: "7e57edca-955c-4caa-ba2f-4c58e3be16a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:34.815378 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.815370    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-podinfo" pod "7e57edca-955c-4caa-ba2f-4c58e3be16a4" (UID: "7e57edca-955c-4caa-ba2f-4c58e3be16a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:34.914860 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.914819    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.914860 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.914861    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.914876    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.914889    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.914937    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef061dbf-d524-4e2b-a234-f14abfd09111-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.914951    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.914962    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.914974    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.914986    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.915001    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x49d6\" (UniqueName: \"kubernetes.io/projected/7e57edca-955c-4caa-ba2f-4c58e3be16a4-kube-api-access-x49d6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.915015    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e57edca-955c-4caa-ba2f-4c58e3be16a4-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.915029    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26lzk\" (UniqueName: \"kubernetes.io/projected/ef061dbf-d524-4e2b-a234-f14abfd09111-kube-api-access-26lzk\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.915076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.915043    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef061dbf-d524-4e2b-a234-f14abfd09111-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:34.926908 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.926872    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"]
Oct 31 07:45:34.931906 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:34.931875    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-5rb6c"]
Oct 31 07:45:35.608251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.608208    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"
Oct 31 07:45:35.608251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.608242    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms" event={"ID":"ef061dbf-d524-4e2b-a234-f14abfd09111","Type":"ContainerDied","Data":"7498e2eacda9c5f883bca434fbb239f9d7447996bd98f3357d54cfce138e5bcf"}
Oct 31 07:45:35.608808 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.608323    5829 scope.go:122] "RemoveContainer" containerID="c2dd6256bcd4746fe4b1311ce9a4dd3b5197803a73e2b194f426eaa93c01da8d"
Oct 31 07:45:35.611759 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.611733    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"
Oct 31 07:45:35.611759 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.611741    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh" event={"ID":"7e57edca-955c-4caa-ba2f-4c58e3be16a4","Type":"ContainerDied","Data":"0f1d534ba015bc63a101eb7502bee4e4d629624f4b61dd8489c0ebb754877884"}
Oct 31 07:45:35.623583 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.623559    5829 scope.go:122] "RemoveContainer" containerID="dc707bdc4b5ae066eb5f7799d82aeecce2f909ccb238d61b37f29c60e5351fb3"
Oct 31 07:45:35.709953 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.709917    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"]
Oct 31 07:45:35.714735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.714700    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-8mwbh"]
Oct 31 07:45:35.746083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.746040    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"]
Oct 31 07:45:35.751953 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:35.751917    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-b1fda4-openshift-default-c9c5b7f4c-p42ms"]
Oct 31 07:45:36.004887 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:36.004810    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" path="/var/lib/kubelet/pods/05a46c7a-6f14-42d9-9012-f4177e2fbad8/volumes"
Oct 31 07:45:36.005349 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:36.005272    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" path="/var/lib/kubelet/pods/7e57edca-955c-4caa-ba2f-4c58e3be16a4/volumes"
Oct 31 07:45:36.005766 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:36.005747    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" path="/var/lib/kubelet/pods/ef061dbf-d524-4e2b-a234-f14abfd09111/volumes"
Oct 31 07:45:36.853835 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:36.853794    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"]
Oct 31 07:45:36.861135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:36.861097    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-3dff3f-qkrhk"]
Oct 31 07:45:36.882576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:36.882525    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq" podUID="b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:37.858921 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:37.858827    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t" podUID="538fd585-48a6-4b18-994e-61767bb14aaa" containerName="payload-processing" containerID="cri-o://be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0" gracePeriod=25
Oct 31 07:45:37.918109 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:37.918067    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r" podUID="821af7a5-dc95-4425-b5b4-5709d5670e5f" containerName="payload-pre-processing" containerID="cri-o://bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969" gracePeriod=25
Oct 31 07:45:37.980341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:37.980115    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5bd411-2ab6-4a3b-bdfa-add465dd3f16" path="/var/lib/kubelet/pods/8a5bd411-2ab6-4a3b-bdfa-add465dd3f16/volumes"
Oct 31 07:45:39.577676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.577629    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"]
Oct 31 07:45:39.578555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578529    5829 cpu_manager.go:414] "containerMap: removing container" podUID="5c52732f-cee9-421a-ad27-422e304db24e" containerName="revoke-keys"
Oct 31 07:45:39.578555 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578554    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c52732f-cee9-421a-ad27-422e304db24e" containerName="revoke-keys"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578572    5829 cpu_manager.go:414] "containerMap: removing container" podUID="df54b1a3-b0e3-4481-b483-250b1136c78d" containerName="payload-processing"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578580    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df54b1a3-b0e3-4481-b483-250b1136c78d" containerName="payload-processing"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578598    5829 cpu_manager.go:414] "containerMap: removing container" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578606    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578623    5829 cpu_manager.go:414] "containerMap: removing container" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578630    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578640    5829 cpu_manager.go:414] "containerMap: removing container" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578648    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578716    5829 cpu_manager.go:414] "containerMap: removing container" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578724    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578740    5829 cpu_manager.go:414] "containerMap: removing container" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main"
Oct 31 07:45:39.578750 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578748    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578763    5829 cpu_manager.go:414] "containerMap: removing container" podUID="eb1f2998-f127-446f-8f12-216bc91e88a7" containerName="revoke-keys"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578770    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1f2998-f127-446f-8f12-216bc91e88a7" containerName="revoke-keys"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578876    5829 cpu_manager.go:414] "containerMap: removing container" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578887    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578944    5829 cpu_manager.go:414] "containerMap: removing container" podUID="874be886-01ff-4e96-be4f-fbe8c0ae7334" containerName="revoke-keys"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.578954    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874be886-01ff-4e96-be4f-fbe8c0ae7334" containerName="revoke-keys"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579153    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="df54b1a3-b0e3-4481-b483-250b1136c78d" containerName="payload-processing"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579172    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2cac2f4-192d-4553-87fb-a0961517fc68" containerName="main"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579185    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579195    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f55dae4-8d1e-4dba-a2d4-b1e791b39157" containerName="main"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579205    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e57edca-955c-4caa-ba2f-4c58e3be16a4" containerName="istio-proxy"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579214    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b33107c-7771-44a0-8600-4ab5d2b8d1e4" containerName="main"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579225    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1f2998-f127-446f-8f12-216bc91e88a7" containerName="revoke-keys"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579236    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a46c7a-6f14-42d9-9012-f4177e2fbad8" containerName="istio-proxy"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579248    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="874be886-01ff-4e96-be4f-fbe8c0ae7334" containerName="revoke-keys"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579261    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy"
Oct 31 07:45:39.579325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.579273    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c52732f-cee9-421a-ad27-422e304db24e" containerName="revoke-keys"
Oct 31 07:45:39.585857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.585830    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.596982 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.596945    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"]
Oct 31 07:45:39.613888 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.613851    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:45:39.614050 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.614019    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:45:39.614371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.614344    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:45:39.614478 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.614404    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:45:39.684115 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.684080    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s5g5\" (UniqueName: \"kubernetes.io/projected/0f9a29cc-5c61-4200-b808-d95bc47fce1e-kube-api-access-5s5g5\") pod \"maas-api-revoke-sub-d1f09f4c-wwqt4\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.684354 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.684138    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f09f4c-wwqt4\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.684354 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.684276    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f09f4c-wwqt4\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.785342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.785307    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s5g5\" (UniqueName: \"kubernetes.io/projected/0f9a29cc-5c61-4200-b808-d95bc47fce1e-kube-api-access-5s5g5\") pod \"maas-api-revoke-sub-d1f09f4c-wwqt4\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.785518 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.785372    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f09f4c-wwqt4\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.785518 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.785417    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f09f4c-wwqt4\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.786110 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.786085    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1f09f4c-wwqt4\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.787756 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.787724    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1f09f4c-wwqt4\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.805235 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.805205    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s5g5\" (UniqueName: \"kubernetes.io/projected/0f9a29cc-5c61-4200-b808-d95bc47fce1e-kube-api-access-5s5g5\") pod \"maas-api-revoke-sub-d1f09f4c-wwqt4\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:39.906935 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.906900    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t" podUID="538fd585-48a6-4b18-994e-61767bb14aaa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:39.906935 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.906916    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r" podUID="821af7a5-dc95-4425-b5b4-5709d5670e5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:39.909117 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:39.909094    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:40.236716 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:45:40.236673    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9a29cc_5c61_4200_b808_d95bc47fce1e.slice/crio-168d17d53a5903ad3ddad936a02717230f62ba8de02117c5c3a2de2ea60fb8f0 WatchSource:0}: Error finding container 168d17d53a5903ad3ddad936a02717230f62ba8de02117c5c3a2de2ea60fb8f0: Status 404 returned error can't find the container with id 168d17d53a5903ad3ddad936a02717230f62ba8de02117c5c3a2de2ea60fb8f0
Oct 31 07:45:40.237039 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:40.237013    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"]
Oct 31 07:45:40.700474 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:40.700416    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4" event={"ID":"0f9a29cc-5c61-4200-b808-d95bc47fce1e","Type":"ContainerStarted","Data":"009097d33939614b56f8ab35d3851c662499b8641bf0e9c0c3da342198320ded"}
Oct 31 07:45:40.700947 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:40.700534    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4" event={"ID":"0f9a29cc-5c61-4200-b808-d95bc47fce1e","Type":"ContainerStarted","Data":"168d17d53a5903ad3ddad936a02717230f62ba8de02117c5c3a2de2ea60fb8f0"}
Oct 31 07:45:40.743680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:40.743625    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4" podStartSLOduration=1.743611867 podStartE2EDuration="1.743611867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:45:40.738721717 +0000 UTC m=+1004.877882306" watchObservedRunningTime="2026-10-31 07:45:40.743611867 +0000 UTC m=+1004.882772455"
Oct 31 07:45:41.710801 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:41.710754    5829 generic.go:356] "Generic (PLEG): container finished" podID="0f9a29cc-5c61-4200-b808-d95bc47fce1e" containerID="009097d33939614b56f8ab35d3851c662499b8641bf0e9c0c3da342198320ded" exitCode=0
Oct 31 07:45:41.711275 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:41.711027    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4" event={"ID":"0f9a29cc-5c61-4200-b808-d95bc47fce1e","Type":"ContainerDied","Data":"009097d33939614b56f8ab35d3851c662499b8641bf0e9c0c3da342198320ded"}
Oct 31 07:45:42.885044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:42.885014    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:42.922363 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:42.922324    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-xks-ca\") pod \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") "
Oct 31 07:45:42.922531 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:42.922438    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9a29cc-5c61-4200-b808-d95bc47fce1e-kube-api-access-5s5g5\" (UniqueName: \"kubernetes.io/projected/0f9a29cc-5c61-4200-b808-d95bc47fce1e-kube-api-access-5s5g5\") pod \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") "
Oct 31 07:45:42.922531 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:42.922487    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-openshift-ca\") pod \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\" (UID: \"0f9a29cc-5c61-4200-b808-d95bc47fce1e\") "
Oct 31 07:45:42.922911 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:42.922848    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-openshift-ca" pod "0f9a29cc-5c61-4200-b808-d95bc47fce1e" (UID: "0f9a29cc-5c61-4200-b808-d95bc47fce1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:42.923099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:42.923069    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:42.925079 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:42.925047    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-xks-ca" pod "0f9a29cc-5c61-4200-b808-d95bc47fce1e" (UID: "0f9a29cc-5c61-4200-b808-d95bc47fce1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:45:42.925079 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:42.925071    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9a29cc-5c61-4200-b808-d95bc47fce1e-kube-api-access-5s5g5" pod "0f9a29cc-5c61-4200-b808-d95bc47fce1e" (UID: "0f9a29cc-5c61-4200-b808-d95bc47fce1e"). InnerVolumeSpecName "kube-api-access-5s5g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:43.024136 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.024034    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9a29cc-5c61-4200-b808-d95bc47fce1e-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:43.024136 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.024075    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s5g5\" (UniqueName: \"kubernetes.io/projected/0f9a29cc-5c61-4200-b808-d95bc47fce1e-kube-api-access-5s5g5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:43.515781 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.515739    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"]
Oct 31 07:45:43.516037 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.516005    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" containerID="cri-o://829dba704a1e6b0758d3be7798cab64d2fc1546d8e2f2b3fedfe1d7cb184106e" gracePeriod=30
Oct 31 07:45:43.528390 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.528355    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"]
Oct 31 07:45:43.528656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.528614    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerName="istio-proxy" containerID="cri-o://748f613b3b4fc57bbfea03542222a043ca71d6853bed7bfc9afc03bda7812114" gracePeriod=30
Oct 31 07:45:43.532965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.532786    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"]
Oct 31 07:45:43.533091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.533007    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy" containerID="cri-o://f7e16fb1d7b4a1c83af4015348128efa12b36e4777411ac7674c5d03081a1d42" gracePeriod=30
Oct 31 07:45:43.732371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.732337    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"
Oct 31 07:45:43.732371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.732354    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4" event={"ID":"0f9a29cc-5c61-4200-b808-d95bc47fce1e","Type":"ContainerDied","Data":"168d17d53a5903ad3ddad936a02717230f62ba8de02117c5c3a2de2ea60fb8f0"}
Oct 31 07:45:43.732600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:43.732447    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="168d17d53a5903ad3ddad936a02717230f62ba8de02117c5c3a2de2ea60fb8f0"
Oct 31 07:45:44.835481 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:44.835445    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"]
Oct 31 07:45:44.835913 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:44.835651    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy" containerID="cri-o://4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d" gracePeriod=30
Oct 31 07:45:44.846686 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:44.846650    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"]
Oct 31 07:45:44.846923 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:44.846892    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy" containerID="cri-o://1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00" gracePeriod=30
Oct 31 07:45:44.851890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:44.851861    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"]
Oct 31 07:45:44.852110 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:44.852076    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy" containerID="cri-o://6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325" gracePeriod=30
Oct 31 07:45:45.036956 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:45.036918    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"]
Oct 31 07:45:45.042191 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:45.042153    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7748bfb0-dzl64"]
Oct 31 07:45:45.975574 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:45.975540    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a518f8-3868-43e6-a909-39ab75e94918" path="/var/lib/kubelet/pods/34a518f8-3868-43e6-a909-39ab75e94918/volumes"
Oct 31 07:45:46.510805 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:46.510766    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:45:46.881229 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:46.881137    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq" podUID="b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:47.448646 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.447689    5829 patch_prober.go:29] interesting pod/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 07:45:47.448646 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.447761    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:45:47.520837 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.520766    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:45:47.588030 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.587987    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-plugins-config-volume\") pod \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\" (UID: \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\") "
Oct 31 07:45:47.588226 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.588160    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-kube-api-access-g4hb9\" (UniqueName: \"kubernetes.io/projected/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-kube-api-access-g4hb9\") pod \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\" (UID: \"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4\") "
Oct 31 07:45:47.588444 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.588419    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-plugins-config-volume" pod "b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" (UID: "b192cb88-7f38-4ea5-b71a-12ecd76fe9c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:47.588701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.588679    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:47.590444 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.590411    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-kube-api-access-g4hb9" pod "b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" (UID: "b192cb88-7f38-4ea5-b71a-12ecd76fe9c4"). InnerVolumeSpecName "kube-api-access-g4hb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:47.690606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.690560    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4hb9\" (UniqueName: \"kubernetes.io/projected/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4-kube-api-access-g4hb9\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:47.771942 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.771852    5829 generic.go:356] "Generic (PLEG): container finished" podID="b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" containerID="ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750" exitCode=0
Oct 31 07:45:47.771942 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.771900    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"
Oct 31 07:45:47.772173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.771939    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq" event={"ID":"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4","Type":"ContainerDied","Data":"ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750"}
Oct 31 07:45:47.772173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.772021    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq" event={"ID":"b192cb88-7f38-4ea5-b71a-12ecd76fe9c4","Type":"ContainerDied","Data":"4a7566914f6bf5a1050d8077e52a45f40a9e87d822fef87d8dff971e4ddb3337"}
Oct 31 07:45:47.772173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.772046    5829 scope.go:122] "RemoveContainer" containerID="ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750"
Oct 31 07:45:47.784394 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.784367    5829 scope.go:122] "RemoveContainer" containerID="ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750"
Oct 31 07:45:47.784687 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:47.784665    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750\": container with ID starting with ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750 not found: ID does not exist" containerID="ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750"
Oct 31 07:45:47.784726 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.784697    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750"} err="failed to get container status \"ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750\": rpc error: code = NotFound desc = could not find container \"ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750\": container with ID starting with ba9e1e68c70d5152e790827beab87e8fddf96a93871914597d57dd93caf32750 not found: ID does not exist"
Oct 31 07:45:47.827130 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.827094    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"]
Oct 31 07:45:47.838650 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.838615    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3cc201-84cbd4d6bd-nm2cq"]
Oct 31 07:45:47.975295 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:47.975256    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" path="/var/lib/kubelet/pods/b192cb88-7f38-4ea5-b71a-12ecd76fe9c4/volumes"
Oct 31 07:45:48.789840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.789750    5829 generic.go:356] "Generic (PLEG): container finished" podID="348fc3b3-343d-4137-9821-c551fc16dd14" containerID="829dba704a1e6b0758d3be7798cab64d2fc1546d8e2f2b3fedfe1d7cb184106e" exitCode=0
Oct 31 07:45:48.789840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.789820    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" event={"ID":"348fc3b3-343d-4137-9821-c551fc16dd14","Type":"ContainerDied","Data":"829dba704a1e6b0758d3be7798cab64d2fc1546d8e2f2b3fedfe1d7cb184106e"}
Oct 31 07:45:48.792188 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.792125    5829 generic.go:356] "Generic (PLEG): container finished" podID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerID="748f613b3b4fc57bbfea03542222a043ca71d6853bed7bfc9afc03bda7812114" exitCode=0
Oct 31 07:45:48.792347 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.792254    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" event={"ID":"7b5c075d-8c75-474b-b71b-15ab59bc2926","Type":"ContainerDied","Data":"748f613b3b4fc57bbfea03542222a043ca71d6853bed7bfc9afc03bda7812114"}
Oct 31 07:45:48.795957 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.795928    5829 generic.go:356] "Generic (PLEG): container finished" podID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerID="f7e16fb1d7b4a1c83af4015348128efa12b36e4777411ac7674c5d03081a1d42" exitCode=0
Oct 31 07:45:48.796090 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.795996    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" event={"ID":"0a750593-3ebd-4a88-b317-8ccd6aaddb8f","Type":"ContainerDied","Data":"f7e16fb1d7b4a1c83af4015348128efa12b36e4777411ac7674c5d03081a1d42"}
Oct 31 07:45:48.855353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.855295    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:48.872930 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.872906    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:45:48.876498 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.876474    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:45:48.907413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907370    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-istio-token\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-istio-token\") pod \"348fc3b3-343d-4137-9821-c551fc16dd14\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") "
Oct 31 07:45:48.907413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907422    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-envoy\") pod \"348fc3b3-343d-4137-9821-c551fc16dd14\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") "
Oct 31 07:45:48.907617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907544    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-credential-socket\") pod \"348fc3b3-343d-4137-9821-c551fc16dd14\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") "
Oct 31 07:45:48.907617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907588    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348fc3b3-343d-4137-9821-c551fc16dd14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348fc3b3-343d-4137-9821-c551fc16dd14-istiod-ca-cert\") pod \"348fc3b3-343d-4137-9821-c551fc16dd14\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") "
Oct 31 07:45:48.907715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907624    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-data\") pod \"348fc3b3-343d-4137-9821-c551fc16dd14\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") "
Oct 31 07:45:48.907715 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907658    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-kube-api-access-5vckw\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-kube-api-access-5vckw\") pod \"348fc3b3-343d-4137-9821-c551fc16dd14\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") "
Oct 31 07:45:48.907816 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907740    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-certs\") pod \"348fc3b3-343d-4137-9821-c551fc16dd14\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") "
Oct 31 07:45:48.907873 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907821    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-credential-socket" pod "348fc3b3-343d-4137-9821-c551fc16dd14" (UID: "348fc3b3-343d-4137-9821-c551fc16dd14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:48.907966 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907920    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-data" pod "348fc3b3-343d-4137-9821-c551fc16dd14" (UID: "348fc3b3-343d-4137-9821-c551fc16dd14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:48.907966 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.907947    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/348fc3b3-343d-4137-9821-c551fc16dd14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348fc3b3-343d-4137-9821-c551fc16dd14-istio-podinfo\") pod \"348fc3b3-343d-4137-9821-c551fc16dd14\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") "
Oct 31 07:45:48.908079 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.908062    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-socket\") pod \"348fc3b3-343d-4137-9821-c551fc16dd14\" (UID: \"348fc3b3-343d-4137-9821-c551fc16dd14\") "
Oct 31 07:45:48.908207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.908124    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-certs" pod "348fc3b3-343d-4137-9821-c551fc16dd14" (UID: "348fc3b3-343d-4137-9821-c551fc16dd14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:48.908372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.908349    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-socket" pod "348fc3b3-343d-4137-9821-c551fc16dd14" (UID: "348fc3b3-343d-4137-9821-c551fc16dd14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:48.908372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.908358    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348fc3b3-343d-4137-9821-c551fc16dd14-istiod-ca-cert" pod "348fc3b3-343d-4137-9821-c551fc16dd14" (UID: "348fc3b3-343d-4137-9821-c551fc16dd14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:48.908705 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.908677    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:48.908705 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.908704    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:48.908828 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.908719    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/348fc3b3-343d-4137-9821-c551fc16dd14-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:48.908828 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.908731    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:48.908828 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.908745    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:48.909963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.909940    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-envoy" pod "348fc3b3-343d-4137-9821-c551fc16dd14" (UID: "348fc3b3-343d-4137-9821-c551fc16dd14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:48.910169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.910150    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-istio-token" pod "348fc3b3-343d-4137-9821-c551fc16dd14" (UID: "348fc3b3-343d-4137-9821-c551fc16dd14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:48.910467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.910443    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/348fc3b3-343d-4137-9821-c551fc16dd14-istio-podinfo" pod "348fc3b3-343d-4137-9821-c551fc16dd14" (UID: "348fc3b3-343d-4137-9821-c551fc16dd14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:48.910554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:48.910522    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-kube-api-access-5vckw" pod "348fc3b3-343d-4137-9821-c551fc16dd14" (UID: "348fc3b3-343d-4137-9821-c551fc16dd14"). InnerVolumeSpecName "kube-api-access-5vckw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:49.009782 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.009399    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-kube-api-access-6f7tn\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-kube-api-access-6f7tn\") pod \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") "
Oct 31 07:45:49.010052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010020    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-podinfo\") pod \"7b5c075d-8c75-474b-b71b-15ab59bc2926\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") "
Oct 31 07:45:49.010220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010095    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-envoy\") pod \"7b5c075d-8c75-474b-b71b-15ab59bc2926\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") "
Oct 31 07:45:49.010220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010123    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-certs\") pod \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") "
Oct 31 07:45:49.010220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010150    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-socket\") pod \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") "
Oct 31 07:45:49.010220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010215    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-data\") pod \"7b5c075d-8c75-474b-b71b-15ab59bc2926\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") "
Oct 31 07:45:49.010562 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010259    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-credential-socket\") pod \"7b5c075d-8c75-474b-b71b-15ab59bc2926\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") "
Oct 31 07:45:49.010562 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010328    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-token\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-token\") pod \"7b5c075d-8c75-474b-b71b-15ab59bc2926\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") "
Oct 31 07:45:49.010562 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010362    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-credential-socket\") pod \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") "
Oct 31 07:45:49.010562 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010392    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-socket\") pod \"7b5c075d-8c75-474b-b71b-15ab59bc2926\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") "
Oct 31 07:45:49.010562 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010429    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-podinfo\") pod \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") "
Oct 31 07:45:49.010562 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010477    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istiod-ca-cert\") pod \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") "
Oct 31 07:45:49.010955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010619    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5c075d-8c75-474b-b71b-15ab59bc2926-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c075d-8c75-474b-b71b-15ab59bc2926-istiod-ca-cert\") pod \"7b5c075d-8c75-474b-b71b-15ab59bc2926\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") "
Oct 31 07:45:49.010955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010653    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-certs\") pod \"7b5c075d-8c75-474b-b71b-15ab59bc2926\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") "
Oct 31 07:45:49.010955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010703    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-token\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-token\") pod \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") "
Oct 31 07:45:49.010955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010741    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-data\") pod \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") "
Oct 31 07:45:49.010955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010775    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-kube-api-access-tb2n4\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-kube-api-access-tb2n4\") pod \"7b5c075d-8c75-474b-b71b-15ab59bc2926\" (UID: \"7b5c075d-8c75-474b-b71b-15ab59bc2926\") "
Oct 31 07:45:49.010955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.010812    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-envoy\") pod \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\" (UID: \"0a750593-3ebd-4a88-b317-8ccd6aaddb8f\") "
Oct 31 07:45:49.012949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.011435    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.012949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.011936    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/348fc3b3-343d-4137-9821-c551fc16dd14-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.012949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.011984    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vckw\" (UniqueName: \"kubernetes.io/projected/348fc3b3-343d-4137-9821-c551fc16dd14-kube-api-access-5vckw\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.012949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.012004    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/348fc3b3-343d-4137-9821-c551fc16dd14-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.012949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.011882    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-credential-socket" pod "0a750593-3ebd-4a88-b317-8ccd6aaddb8f" (UID: "0a750593-3ebd-4a88-b317-8ccd6aaddb8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.012949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.012138    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-socket" pod "7b5c075d-8c75-474b-b71b-15ab59bc2926" (UID: "7b5c075d-8c75-474b-b71b-15ab59bc2926"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.013987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.013173    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istiod-ca-cert" pod "0a750593-3ebd-4a88-b317-8ccd6aaddb8f" (UID: "0a750593-3ebd-4a88-b317-8ccd6aaddb8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:49.013987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.013542    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5c075d-8c75-474b-b71b-15ab59bc2926-istiod-ca-cert" pod "7b5c075d-8c75-474b-b71b-15ab59bc2926" (UID: "7b5c075d-8c75-474b-b71b-15ab59bc2926"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:49.013987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.013790    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-certs" pod "7b5c075d-8c75-474b-b71b-15ab59bc2926" (UID: "7b5c075d-8c75-474b-b71b-15ab59bc2926"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.014135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.014065    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-data" pod "7b5c075d-8c75-474b-b71b-15ab59bc2926" (UID: "7b5c075d-8c75-474b-b71b-15ab59bc2926"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.014315 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.014257    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-socket" pod "0a750593-3ebd-4a88-b317-8ccd6aaddb8f" (UID: "0a750593-3ebd-4a88-b317-8ccd6aaddb8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.014634 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.014472    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-data" pod "0a750593-3ebd-4a88-b317-8ccd6aaddb8f" (UID: "0a750593-3ebd-4a88-b317-8ccd6aaddb8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.015044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.014779    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-credential-socket" pod "7b5c075d-8c75-474b-b71b-15ab59bc2926" (UID: "7b5c075d-8c75-474b-b71b-15ab59bc2926"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.015929 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.015891    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-podinfo" pod "0a750593-3ebd-4a88-b317-8ccd6aaddb8f" (UID: "0a750593-3ebd-4a88-b317-8ccd6aaddb8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:49.016181 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.016152    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-certs" pod "0a750593-3ebd-4a88-b317-8ccd6aaddb8f" (UID: "0a750593-3ebd-4a88-b317-8ccd6aaddb8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.016688 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.016552    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-podinfo" pod "7b5c075d-8c75-474b-b71b-15ab59bc2926" (UID: "7b5c075d-8c75-474b-b71b-15ab59bc2926"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:49.016688 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.016641    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-kube-api-access-6f7tn" pod "0a750593-3ebd-4a88-b317-8ccd6aaddb8f" (UID: "0a750593-3ebd-4a88-b317-8ccd6aaddb8f"). InnerVolumeSpecName "kube-api-access-6f7tn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:49.016843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.016773    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-envoy" pod "0a750593-3ebd-4a88-b317-8ccd6aaddb8f" (UID: "0a750593-3ebd-4a88-b317-8ccd6aaddb8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.016912 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.016855    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-token" pod "0a750593-3ebd-4a88-b317-8ccd6aaddb8f" (UID: "0a750593-3ebd-4a88-b317-8ccd6aaddb8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:49.018144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.018115    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-kube-api-access-tb2n4" pod "7b5c075d-8c75-474b-b71b-15ab59bc2926" (UID: "7b5c075d-8c75-474b-b71b-15ab59bc2926"). InnerVolumeSpecName "kube-api-access-tb2n4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:49.018242 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.018182    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-envoy" pod "7b5c075d-8c75-474b-b71b-15ab59bc2926" (UID: "7b5c075d-8c75-474b-b71b-15ab59bc2926"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:49.018327 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.018265    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-token" pod "7b5c075d-8c75-474b-b71b-15ab59bc2926" (UID: "7b5c075d-8c75-474b-b71b-15ab59bc2926"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:49.113876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113774    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f7tn\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-kube-api-access-6f7tn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.113876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113847    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.113876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113867    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.113876 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113878    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113887    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113895    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113904    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113915    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113924    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113932    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113943    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113951    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113960    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c075d-8c75-474b-b71b-15ab59bc2926-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113969    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5c075d-8c75-474b-b71b-15ab59bc2926-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113977    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113985    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.113994    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb2n4\" (UniqueName: \"kubernetes.io/projected/7b5c075d-8c75-474b-b71b-15ab59bc2926-kube-api-access-tb2n4\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.114126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.114002    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a750593-3ebd-4a88-b317-8ccd6aaddb8f-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:49.807067 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.806970    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"
Oct 31 07:45:49.807067 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.807005    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv" event={"ID":"7b5c075d-8c75-474b-b71b-15ab59bc2926","Type":"ContainerDied","Data":"36de0b6eefada1b54f0d497b1fed8590a5a300dc07ddd61138b6e3c857b739d3"}
Oct 31 07:45:49.807619 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.807070    5829 scope.go:122] "RemoveContainer" containerID="748f613b3b4fc57bbfea03542222a043ca71d6853bed7bfc9afc03bda7812114"
Oct 31 07:45:49.808915 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.808890    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"
Oct 31 07:45:49.808915 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.808903    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft" event={"ID":"0a750593-3ebd-4a88-b317-8ccd6aaddb8f","Type":"ContainerDied","Data":"1bb7b9017ef3a16b0c29bc04810816466437e81e534a8c25c3e8c8c6a48400f1"}
Oct 31 07:45:49.810909 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.810887    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"
Oct 31 07:45:49.810909 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.810904    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6" event={"ID":"348fc3b3-343d-4137-9821-c551fc16dd14","Type":"ContainerDied","Data":"7862ddb79784476f95ca3fc3c4829c03730c0b586c6059165a96d2488c715139"}
Oct 31 07:45:49.820178 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.820151    5829 scope.go:122] "RemoveContainer" containerID="f7e16fb1d7b4a1c83af4015348128efa12b36e4777411ac7674c5d03081a1d42"
Oct 31 07:45:49.832195 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.832174    5829 scope.go:122] "RemoveContainer" containerID="829dba704a1e6b0758d3be7798cab64d2fc1546d8e2f2b3fedfe1d7cb184106e"
Oct 31 07:45:49.894166 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.894136    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"]
Oct 31 07:45:49.906642 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.906598    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-lnrgv"]
Oct 31 07:45:49.907717 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.907680    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t" podUID="538fd585-48a6-4b18-994e-61767bb14aaa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:49.907833 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.907771    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r" podUID="821af7a5-dc95-4425-b5b4-5709d5670e5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:45:49.944516 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.944485    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"]
Oct 31 07:45:49.949408 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.949380    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-svcft"]
Oct 31 07:45:49.985450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.984122    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" path="/var/lib/kubelet/pods/0a750593-3ebd-4a88-b317-8ccd6aaddb8f/volumes"
Oct 31 07:45:49.985450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.984931    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" path="/var/lib/kubelet/pods/7b5c075d-8c75-474b-b71b-15ab59bc2926/volumes"
Oct 31 07:45:49.991706 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:49.991675    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"]
Oct 31 07:45:50.000897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.000859    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3dff3f-openshift-default-789d5dc677-sxbw6"]
Oct 31 07:45:50.132718 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.132687    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:45:50.144510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.144485    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:45:50.196101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.196075    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:45:50.240592 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.240558    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-data\") pod \"07174d73-53ba-408f-bf2f-6f4273be8fe3\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") "
Oct 31 07:45:50.240868 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.240851    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-credential-socket\") pod \"07174d73-53ba-408f-bf2f-6f4273be8fe3\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") "
Oct 31 07:45:50.240976 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.240892    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-envoy\") pod \"07174d73-53ba-408f-bf2f-6f4273be8fe3\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") "
Oct 31 07:45:50.240976 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.240959    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-envoy\") pod \"965ba733-f75b-4e9b-b486-443bdea104e2\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") "
Oct 31 07:45:50.241084 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241002    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-certs\") pod \"07174d73-53ba-408f-bf2f-6f4273be8fe3\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") "
Oct 31 07:45:50.241084 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241053    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-token\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-token\") pod \"07174d73-53ba-408f-bf2f-6f4273be8fe3\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") "
Oct 31 07:45:50.241190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241098    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-data" pod "07174d73-53ba-408f-bf2f-6f4273be8fe3" (UID: "07174d73-53ba-408f-bf2f-6f4273be8fe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.241190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241124    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-socket\") pod \"965ba733-f75b-4e9b-b486-443bdea104e2\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") "
Oct 31 07:45:50.241190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241156    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-credential-socket" pod "07174d73-53ba-408f-bf2f-6f4273be8fe3" (UID: "07174d73-53ba-408f-bf2f-6f4273be8fe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.241367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241190    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/965ba733-f75b-4e9b-b486-443bdea104e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965ba733-f75b-4e9b-b486-443bdea104e2-istio-podinfo\") pod \"965ba733-f75b-4e9b-b486-443bdea104e2\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") "
Oct 31 07:45:50.241367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241231    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-podinfo\") pod \"07174d73-53ba-408f-bf2f-6f4273be8fe3\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") "
Oct 31 07:45:50.241367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241262    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-socket\") pod \"07174d73-53ba-408f-bf2f-6f4273be8fe3\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") "
Oct 31 07:45:50.241367 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241334    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-credential-socket\") pod \"965ba733-f75b-4e9b-b486-443bdea104e2\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") "
Oct 31 07:45:50.241564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241390    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07174d73-53ba-408f-bf2f-6f4273be8fe3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07174d73-53ba-408f-bf2f-6f4273be8fe3-istiod-ca-cert\") pod \"07174d73-53ba-408f-bf2f-6f4273be8fe3\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") "
Oct 31 07:45:50.241564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241439    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-data\") pod \"965ba733-f75b-4e9b-b486-443bdea104e2\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") "
Oct 31 07:45:50.241564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241497    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/965ba733-f75b-4e9b-b486-443bdea104e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965ba733-f75b-4e9b-b486-443bdea104e2-istiod-ca-cert\") pod \"965ba733-f75b-4e9b-b486-443bdea104e2\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") "
Oct 31 07:45:50.241564 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241547    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-kube-api-access-j2m8q\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-kube-api-access-j2m8q\") pod \"07174d73-53ba-408f-bf2f-6f4273be8fe3\" (UID: \"07174d73-53ba-408f-bf2f-6f4273be8fe3\") "
Oct 31 07:45:50.241755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241596    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-istio-token\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-istio-token\") pod \"965ba733-f75b-4e9b-b486-443bdea104e2\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") "
Oct 31 07:45:50.241755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241646    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-kube-api-access-px6rs\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-kube-api-access-px6rs\") pod \"965ba733-f75b-4e9b-b486-443bdea104e2\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") "
Oct 31 07:45:50.241755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.241687    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-certs\") pod \"965ba733-f75b-4e9b-b486-443bdea104e2\" (UID: \"965ba733-f75b-4e9b-b486-443bdea104e2\") "
Oct 31 07:45:50.242309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.242121    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-certs" pod "07174d73-53ba-408f-bf2f-6f4273be8fe3" (UID: "07174d73-53ba-408f-bf2f-6f4273be8fe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.242309 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.242247    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.242799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.242497    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.242799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.242532    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.242799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.242343    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-certs" pod "965ba733-f75b-4e9b-b486-443bdea104e2" (UID: "965ba733-f75b-4e9b-b486-443bdea104e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.243479 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.243453    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07174d73-53ba-408f-bf2f-6f4273be8fe3-istiod-ca-cert" pod "07174d73-53ba-408f-bf2f-6f4273be8fe3" (UID: "07174d73-53ba-408f-bf2f-6f4273be8fe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:50.243681 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.243653    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-envoy" pod "965ba733-f75b-4e9b-b486-443bdea104e2" (UID: "965ba733-f75b-4e9b-b486-443bdea104e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.243955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.243931    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-socket" pod "965ba733-f75b-4e9b-b486-443bdea104e2" (UID: "965ba733-f75b-4e9b-b486-443bdea104e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.244256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.244226    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-token" pod "07174d73-53ba-408f-bf2f-6f4273be8fe3" (UID: "07174d73-53ba-408f-bf2f-6f4273be8fe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:50.244256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.244235    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-socket" pod "07174d73-53ba-408f-bf2f-6f4273be8fe3" (UID: "07174d73-53ba-408f-bf2f-6f4273be8fe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.244504 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.244481    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-credential-socket" pod "965ba733-f75b-4e9b-b486-443bdea104e2" (UID: "965ba733-f75b-4e9b-b486-443bdea104e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.244685 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.244662    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-data" pod "965ba733-f75b-4e9b-b486-443bdea104e2" (UID: "965ba733-f75b-4e9b-b486-443bdea104e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.244961 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.244920    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/965ba733-f75b-4e9b-b486-443bdea104e2-istiod-ca-cert" pod "965ba733-f75b-4e9b-b486-443bdea104e2" (UID: "965ba733-f75b-4e9b-b486-443bdea104e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:50.245192 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.245173    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-istio-token" pod "965ba733-f75b-4e9b-b486-443bdea104e2" (UID: "965ba733-f75b-4e9b-b486-443bdea104e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:50.245790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.245768    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-podinfo" pod "07174d73-53ba-408f-bf2f-6f4273be8fe3" (UID: "07174d73-53ba-408f-bf2f-6f4273be8fe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:50.246546 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.246523    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-envoy" pod "07174d73-53ba-408f-bf2f-6f4273be8fe3" (UID: "07174d73-53ba-408f-bf2f-6f4273be8fe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.246630 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.246612    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-kube-api-access-px6rs" pod "965ba733-f75b-4e9b-b486-443bdea104e2" (UID: "965ba733-f75b-4e9b-b486-443bdea104e2"). InnerVolumeSpecName "kube-api-access-px6rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:50.246724 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.246710    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/965ba733-f75b-4e9b-b486-443bdea104e2-istio-podinfo" pod "965ba733-f75b-4e9b-b486-443bdea104e2" (UID: "965ba733-f75b-4e9b-b486-443bdea104e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:50.247139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.247123    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-kube-api-access-j2m8q" pod "07174d73-53ba-408f-bf2f-6f4273be8fe3" (UID: "07174d73-53ba-408f-bf2f-6f4273be8fe3"). InnerVolumeSpecName "kube-api-access-j2m8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:50.343192 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343148    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-credential-socket\") pod \"8a57dee9-c39b-4039-b990-889f7ecb0d96\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") "
Oct 31 07:45:50.343383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343204    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a57dee9-c39b-4039-b990-889f7ecb0d96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a57dee9-c39b-4039-b990-889f7ecb0d96-istiod-ca-cert\") pod \"8a57dee9-c39b-4039-b990-889f7ecb0d96\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") "
Oct 31 07:45:50.343383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343323    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-token\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-token\") pod \"8a57dee9-c39b-4039-b990-889f7ecb0d96\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") "
Oct 31 07:45:50.343503 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343403    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-data\") pod \"8a57dee9-c39b-4039-b990-889f7ecb0d96\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") "
Oct 31 07:45:50.343503 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343458    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-podinfo\") pod \"8a57dee9-c39b-4039-b990-889f7ecb0d96\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") "
Oct 31 07:45:50.343626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343511    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-credential-socket" pod "8a57dee9-c39b-4039-b990-889f7ecb0d96" (UID: "8a57dee9-c39b-4039-b990-889f7ecb0d96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.343626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343540    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-kube-api-access-nqs92\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-kube-api-access-nqs92\") pod \"8a57dee9-c39b-4039-b990-889f7ecb0d96\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") "
Oct 31 07:45:50.343736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343663    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-certs\") pod \"8a57dee9-c39b-4039-b990-889f7ecb0d96\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") "
Oct 31 07:45:50.343736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343705    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-socket\") pod \"8a57dee9-c39b-4039-b990-889f7ecb0d96\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") "
Oct 31 07:45:50.343736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343712    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-data" pod "8a57dee9-c39b-4039-b990-889f7ecb0d96" (UID: "8a57dee9-c39b-4039-b990-889f7ecb0d96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.343890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343733    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a57dee9-c39b-4039-b990-889f7ecb0d96-istiod-ca-cert" pod "8a57dee9-c39b-4039-b990-889f7ecb0d96" (UID: "8a57dee9-c39b-4039-b990-889f7ecb0d96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:50.343890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.343751    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-envoy\") pod \"8a57dee9-c39b-4039-b990-889f7ecb0d96\" (UID: \"8a57dee9-c39b-4039-b990-889f7ecb0d96\") "
Oct 31 07:45:50.344030 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344011    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-certs" pod "8a57dee9-c39b-4039-b990-889f7ecb0d96" (UID: "8a57dee9-c39b-4039-b990-889f7ecb0d96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.344311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344038    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-socket" pod "8a57dee9-c39b-4039-b990-889f7ecb0d96" (UID: "8a57dee9-c39b-4039-b990-889f7ecb0d96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.344434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344416    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.344655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344630    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.344735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344655    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.344735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344670    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.344735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344686    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/965ba733-f75b-4e9b-b486-443bdea104e2-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.344735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344700    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.344735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344713    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.344735 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344727    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344741    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07174d73-53ba-408f-bf2f-6f4273be8fe3-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344754    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344767    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/965ba733-f75b-4e9b-b486-443bdea104e2-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344779    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344792    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2m8q\" (UniqueName: \"kubernetes.io/projected/07174d73-53ba-408f-bf2f-6f4273be8fe3-kube-api-access-j2m8q\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344807    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344821    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344835    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6rs\" (UniqueName: \"kubernetes.io/projected/965ba733-f75b-4e9b-b486-443bdea104e2-kube-api-access-px6rs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344849    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/965ba733-f75b-4e9b-b486-443bdea104e2-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344863    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344879    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a57dee9-c39b-4039-b990-889f7ecb0d96-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.345046 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.344893    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07174d73-53ba-408f-bf2f-6f4273be8fe3-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.346682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.346642    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-podinfo" pod "8a57dee9-c39b-4039-b990-889f7ecb0d96" (UID: "8a57dee9-c39b-4039-b990-889f7ecb0d96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:45:50.346682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.346660    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-kube-api-access-nqs92" pod "8a57dee9-c39b-4039-b990-889f7ecb0d96" (UID: "8a57dee9-c39b-4039-b990-889f7ecb0d96"). InnerVolumeSpecName "kube-api-access-nqs92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:50.346862 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.346686    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-envoy" pod "8a57dee9-c39b-4039-b990-889f7ecb0d96" (UID: "8a57dee9-c39b-4039-b990-889f7ecb0d96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:45:50.348209 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.348187    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-token" pod "8a57dee9-c39b-4039-b990-889f7ecb0d96" (UID: "8a57dee9-c39b-4039-b990-889f7ecb0d96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:50.446463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.446425    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.446463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.446456    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.446463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.446467    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqs92\" (UniqueName: \"kubernetes.io/projected/8a57dee9-c39b-4039-b990-889f7ecb0d96-kube-api-access-nqs92\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.446721 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.446477    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a57dee9-c39b-4039-b990-889f7ecb0d96-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:50.821948 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.821857    5829 generic.go:356] "Generic (PLEG): container finished" podID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerID="1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00" exitCode=0
Oct 31 07:45:50.821948 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.821935    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"
Oct 31 07:45:50.822515 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.821939    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" event={"ID":"8a57dee9-c39b-4039-b990-889f7ecb0d96","Type":"ContainerDied","Data":"1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00"}
Oct 31 07:45:50.822515 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.822024    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76" event={"ID":"8a57dee9-c39b-4039-b990-889f7ecb0d96","Type":"ContainerDied","Data":"d4f619f2454a4b1bd0b4f6751a77ad2abee1961734ce52483be2922323cdf6c8"}
Oct 31 07:45:50.822515 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.822048    5829 scope.go:122] "RemoveContainer" containerID="1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00"
Oct 31 07:45:50.823777 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.823750    5829 generic.go:356] "Generic (PLEG): container finished" podID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerID="6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325" exitCode=0
Oct 31 07:45:50.823864 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.823823    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"
Oct 31 07:45:50.823938 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.823845    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" event={"ID":"07174d73-53ba-408f-bf2f-6f4273be8fe3","Type":"ContainerDied","Data":"6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325"}
Oct 31 07:45:50.823997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.823927    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k" event={"ID":"07174d73-53ba-408f-bf2f-6f4273be8fe3","Type":"ContainerDied","Data":"84a91ca8d02262bd330de692cd55c4de10a03c8817036f740d465918f486660e"}
Oct 31 07:45:50.825944 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.825920    5829 generic.go:356] "Generic (PLEG): container finished" podID="965ba733-f75b-4e9b-b486-443bdea104e2" containerID="4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d" exitCode=0
Oct 31 07:45:50.826042 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.826013    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"
Oct 31 07:45:50.826099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.826057    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" event={"ID":"965ba733-f75b-4e9b-b486-443bdea104e2","Type":"ContainerDied","Data":"4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d"}
Oct 31 07:45:50.826166 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.826099    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd" event={"ID":"965ba733-f75b-4e9b-b486-443bdea104e2","Type":"ContainerDied","Data":"9a143de8b8c140e0874d9ea0b4dff550c728e02fa985581cc448823776fc4f9a"}
Oct 31 07:45:50.835166 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.835142    5829 scope.go:122] "RemoveContainer" containerID="1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00"
Oct 31 07:45:50.835712 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:50.835680    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00\": container with ID starting with 1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00 not found: ID does not exist" containerID="1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00"
Oct 31 07:45:50.835820 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.835749    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00"} err="failed to get container status \"1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00\": rpc error: code = NotFound desc = could not find container \"1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00\": container with ID starting with 1f60fa30479ef94dc21ad4e0dea41dcb0004f3f74bac25519271c894cb0d9e00 not found: ID does not exist"
Oct 31 07:45:50.835820 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.835786    5829 scope.go:122] "RemoveContainer" containerID="6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325"
Oct 31 07:45:50.849166 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.849135    5829 scope.go:122] "RemoveContainer" containerID="6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325"
Oct 31 07:45:50.849546 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:50.849523    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325\": container with ID starting with 6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325 not found: ID does not exist" containerID="6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325"
Oct 31 07:45:50.849649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.849552    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325"} err="failed to get container status \"6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325\": rpc error: code = NotFound desc = could not find container \"6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325\": container with ID starting with 6780da4d598ca6793d337c046a915ef08a3b269188da2236ec747e99b81f1325 not found: ID does not exist"
Oct 31 07:45:50.849649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.849571    5829 scope.go:122] "RemoveContainer" containerID="4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d"
Oct 31 07:45:50.860201 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.860178    5829 scope.go:122] "RemoveContainer" containerID="4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d"
Oct 31 07:45:50.860527 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:50.860505    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d\": container with ID starting with 4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d not found: ID does not exist" containerID="4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d"
Oct 31 07:45:50.860621 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.860535    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d"} err="failed to get container status \"4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d\": rpc error: code = NotFound desc = could not find container \"4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d\": container with ID starting with 4fa6097b640ea103206d700a06d964f63fb17940bcfbe934c060e19db3ecff9d not found: ID does not exist"
Oct 31 07:45:50.879092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.879060    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"]
Oct 31 07:45:50.893727 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.893687    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-hwg76"]
Oct 31 07:45:50.949840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.949804    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"]
Oct 31 07:45:50.954793 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.954761    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-kj28k"]
Oct 31 07:45:50.987529 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.987492    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"]
Oct 31 07:45:50.994963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:50.994918    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3cc201-openshift-default-ddf9f7b95-zchqd"]
Oct 31 07:45:51.975139 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:51.975101    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" path="/var/lib/kubelet/pods/07174d73-53ba-408f-bf2f-6f4273be8fe3/volumes"
Oct 31 07:45:51.975595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:51.975581    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" path="/var/lib/kubelet/pods/348fc3b3-343d-4137-9821-c551fc16dd14/volumes"
Oct 31 07:45:51.975993 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:51.975981    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" path="/var/lib/kubelet/pods/8a57dee9-c39b-4039-b990-889f7ecb0d96/volumes"
Oct 31 07:45:51.976741 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:51.976722    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" path="/var/lib/kubelet/pods/965ba733-f75b-4e9b-b486-443bdea104e2/volumes"
Oct 31 07:45:57.029491 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:57.029452    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"]
Oct 31 07:45:57.042171 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:57.042137    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a79e028f-vb6md"]
Oct 31 07:45:57.171014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:57.170974    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"]
Oct 31 07:45:57.174946 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:57.174918    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4794f5-fxvb2"]
Oct 31 07:45:57.975016 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:57.974982    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8" path="/var/lib/kubelet/pods/9ac76004-7ba5-4b5c-9fa8-a967cf0b17e8/volumes"
Oct 31 07:45:57.975527 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:57.975513    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1f2998-f127-446f-8f12-216bc91e88a7" path="/var/lib/kubelet/pods/eb1f2998-f127-446f-8f12-216bc91e88a7/volumes"
Oct 31 07:45:58.172528 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.172493    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:45:58.234294 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.234250    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:45:58.344062 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.344027    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538fd585-48a6-4b18-994e-61767bb14aaa-kube-api-access-tq4t2\" (UniqueName: \"kubernetes.io/projected/538fd585-48a6-4b18-994e-61767bb14aaa-kube-api-access-tq4t2\") pod \"538fd585-48a6-4b18-994e-61767bb14aaa\" (UID: \"538fd585-48a6-4b18-994e-61767bb14aaa\") "
Oct 31 07:45:58.344220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.344094    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538fd585-48a6-4b18-994e-61767bb14aaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538fd585-48a6-4b18-994e-61767bb14aaa-plugins-config-volume\") pod \"538fd585-48a6-4b18-994e-61767bb14aaa\" (UID: \"538fd585-48a6-4b18-994e-61767bb14aaa\") "
Oct 31 07:45:58.344314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.344275    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821af7a5-dc95-4425-b5b4-5709d5670e5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821af7a5-dc95-4425-b5b4-5709d5670e5f-plugins-config-volume\") pod \"821af7a5-dc95-4425-b5b4-5709d5670e5f\" (UID: \"821af7a5-dc95-4425-b5b4-5709d5670e5f\") "
Oct 31 07:45:58.344362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.344337    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821af7a5-dc95-4425-b5b4-5709d5670e5f-kube-api-access-zb4m5\" (UniqueName: \"kubernetes.io/projected/821af7a5-dc95-4425-b5b4-5709d5670e5f-kube-api-access-zb4m5\") pod \"821af7a5-dc95-4425-b5b4-5709d5670e5f\" (UID: \"821af7a5-dc95-4425-b5b4-5709d5670e5f\") "
Oct 31 07:45:58.344543 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.344511    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538fd585-48a6-4b18-994e-61767bb14aaa-plugins-config-volume" pod "538fd585-48a6-4b18-994e-61767bb14aaa" (UID: "538fd585-48a6-4b18-994e-61767bb14aaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:58.344732 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.344703    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821af7a5-dc95-4425-b5b4-5709d5670e5f-plugins-config-volume" pod "821af7a5-dc95-4425-b5b4-5709d5670e5f" (UID: "821af7a5-dc95-4425-b5b4-5709d5670e5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:45:58.344804 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.344749    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/538fd585-48a6-4b18-994e-61767bb14aaa-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:58.346266 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.346243    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538fd585-48a6-4b18-994e-61767bb14aaa-kube-api-access-tq4t2" pod "538fd585-48a6-4b18-994e-61767bb14aaa" (UID: "538fd585-48a6-4b18-994e-61767bb14aaa"). InnerVolumeSpecName "kube-api-access-tq4t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:58.346471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.346451    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821af7a5-dc95-4425-b5b4-5709d5670e5f-kube-api-access-zb4m5" pod "821af7a5-dc95-4425-b5b4-5709d5670e5f" (UID: "821af7a5-dc95-4425-b5b4-5709d5670e5f"). InnerVolumeSpecName "kube-api-access-zb4m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:45:58.446093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.446005    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/821af7a5-dc95-4425-b5b4-5709d5670e5f-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:58.446093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.446039    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb4m5\" (UniqueName: \"kubernetes.io/projected/821af7a5-dc95-4425-b5b4-5709d5670e5f-kube-api-access-zb4m5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:58.446093 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.446050    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq4t2\" (UniqueName: \"kubernetes.io/projected/538fd585-48a6-4b18-994e-61767bb14aaa-kube-api-access-tq4t2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:45:58.922610 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.922569    5829 generic.go:356] "Generic (PLEG): container finished" podID="821af7a5-dc95-4425-b5b4-5709d5670e5f" containerID="bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969" exitCode=0
Oct 31 07:45:58.922829 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.922643    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"
Oct 31 07:45:58.922829 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.922661    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r" event={"ID":"821af7a5-dc95-4425-b5b4-5709d5670e5f","Type":"ContainerDied","Data":"bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969"}
Oct 31 07:45:58.922829 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.922752    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r" event={"ID":"821af7a5-dc95-4425-b5b4-5709d5670e5f","Type":"ContainerDied","Data":"bf38f9ef29f91f0adf7322508181217b9cd523277b2a3da639b20f7e556afed5"}
Oct 31 07:45:58.922829 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.922788    5829 scope.go:122] "RemoveContainer" containerID="bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969"
Oct 31 07:45:58.927615 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.927559    5829 generic.go:356] "Generic (PLEG): container finished" podID="538fd585-48a6-4b18-994e-61767bb14aaa" containerID="be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0" exitCode=0
Oct 31 07:45:58.927763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.927623    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t" event={"ID":"538fd585-48a6-4b18-994e-61767bb14aaa","Type":"ContainerDied","Data":"be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0"}
Oct 31 07:45:58.927763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.927643    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"
Oct 31 07:45:58.927763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.927673    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t" event={"ID":"538fd585-48a6-4b18-994e-61767bb14aaa","Type":"ContainerDied","Data":"9468485e50529dbf10fd9bee5cf7b409b3c640e4d6ec0d9f3be40a86277bb4a9"}
Oct 31 07:45:58.935441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.935415    5829 scope.go:122] "RemoveContainer" containerID="bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969"
Oct 31 07:45:58.935774 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:58.935754    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969\": container with ID starting with bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969 not found: ID does not exist" containerID="bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969"
Oct 31 07:45:58.935831 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.935783    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969"} err="failed to get container status \"bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969\": rpc error: code = NotFound desc = could not find container \"bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969\": container with ID starting with bacf49f8d5a79f132d284d5570fb776d72f6e48ab4760f479bea3bb86dd86969 not found: ID does not exist"
Oct 31 07:45:58.935831 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.935802    5829 scope.go:122] "RemoveContainer" containerID="be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0"
Oct 31 07:45:58.946189 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.946161    5829 scope.go:122] "RemoveContainer" containerID="be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0"
Oct 31 07:45:58.946565 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:45:58.946542    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0\": container with ID starting with be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0 not found: ID does not exist" containerID="be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0"
Oct 31 07:45:58.946626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.946576    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0"} err="failed to get container status \"be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0\": rpc error: code = NotFound desc = could not find container \"be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0\": container with ID starting with be8e60b1b43a75d46d5c4a655451accb807fd009a32c393857b91f250facfbc0 not found: ID does not exist"
Oct 31 07:45:58.978113 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.978082    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"]
Oct 31 07:45:58.982370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:58.982344    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4794f5-6f9547ffc8-snx8t"]
Oct 31 07:45:59.014386 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:59.014351    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"]
Oct 31 07:45:59.018376 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:59.018351    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4794f5-79484d54dc-klw4r"]
Oct 31 07:45:59.974896 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:59.974862    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538fd585-48a6-4b18-994e-61767bb14aaa" path="/var/lib/kubelet/pods/538fd585-48a6-4b18-994e-61767bb14aaa/volumes"
Oct 31 07:45:59.975293 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:45:59.975259    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821af7a5-dc95-4425-b5b4-5709d5670e5f" path="/var/lib/kubelet/pods/821af7a5-dc95-4425-b5b4-5709d5670e5f/volumes"
Oct 31 07:46:02.034827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:02.034793    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"]
Oct 31 07:46:02.047650 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:02.047615    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-137733a5-xrrnc"]
Oct 31 07:46:03.974207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:03.974168    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de922643-08e7-4c2a-9e9a-65a67f902d10" path="/var/lib/kubelet/pods/de922643-08e7-4c2a-9e9a-65a67f902d10/volumes"
Oct 31 07:46:06.753328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:06.753222    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:46:07.827304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:07.826621    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"]
Oct 31 07:46:11.028242 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:11.028205    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"]
Oct 31 07:46:11.041790 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:11.041759    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d24c28d0-88b2b"]
Oct 31 07:46:11.974404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:11.974367    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13831a00-7448-4583-b6cf-1ed17e2ddaec" path="/var/lib/kubelet/pods/13831a00-7448-4583-b6cf-1ed17e2ddaec/volumes"
Oct 31 07:46:13.028052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:13.028015    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"]
Oct 31 07:46:13.041249 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:13.041217    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb4a57ce-zvnb7"]
Oct 31 07:46:13.544160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:13.544126    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"]
Oct 31 07:46:13.975398 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:13.975362    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdf75c6b-480f-48be-a51b-f07000cd37df" path="/var/lib/kubelet/pods/cdf75c6b-480f-48be-a51b-f07000cd37df/volumes"
Oct 31 07:46:14.722525 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.722467    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"]
Oct 31 07:46:14.723244 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723219    5829 cpu_manager.go:414] "containerMap: removing container" podUID="b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" containerName="payload-processing"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723244    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" containerName="payload-processing"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723295    5829 cpu_manager.go:414] "containerMap: removing container" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723303    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723326    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723334    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723351    5829 cpu_manager.go:414] "containerMap: removing container" podUID="0f9a29cc-5c61-4200-b808-d95bc47fce1e" containerName="revoke-keys"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723358    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9a29cc-5c61-4200-b808-d95bc47fce1e" containerName="revoke-keys"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723373    5829 cpu_manager.go:414] "containerMap: removing container" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723380    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723391    5829 cpu_manager.go:414] "containerMap: removing container" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723399    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef061dbf-d524-4e2b-a234-f14abfd09111" containerName="istio-proxy"
Oct 31 07:46:14.723405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723411    5829 cpu_manager.go:414] "containerMap: removing container" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723419    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723438    5829 cpu_manager.go:414] "containerMap: removing container" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723446    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723461    5829 cpu_manager.go:414] "containerMap: removing container" podUID="538fd585-48a6-4b18-994e-61767bb14aaa" containerName="payload-processing"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723468    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538fd585-48a6-4b18-994e-61767bb14aaa" containerName="payload-processing"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723481    5829 cpu_manager.go:414] "containerMap: removing container" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723487    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723513    5829 cpu_manager.go:414] "containerMap: removing container" podUID="821af7a5-dc95-4425-b5b4-5709d5670e5f" containerName="payload-pre-processing"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723523    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821af7a5-dc95-4425-b5b4-5709d5670e5f" containerName="payload-pre-processing"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723662    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="821af7a5-dc95-4425-b5b4-5709d5670e5f" containerName="payload-pre-processing"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723678    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="b192cb88-7f38-4ea5-b71a-12ecd76fe9c4" containerName="payload-processing"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723694    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="348fc3b3-343d-4137-9821-c551fc16dd14" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723705    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a57dee9-c39b-4039-b990-889f7ecb0d96" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723714    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9a29cc-5c61-4200-b808-d95bc47fce1e" containerName="revoke-keys"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723723    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="07174d73-53ba-408f-bf2f-6f4273be8fe3" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723732    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5c075d-8c75-474b-b71b-15ab59bc2926" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723743    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a750593-3ebd-4a88-b317-8ccd6aaddb8f" containerName="istio-proxy"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723752    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="538fd585-48a6-4b18-994e-61767bb14aaa" containerName="payload-processing"
Oct 31 07:46:14.723980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.723762    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="965ba733-f75b-4e9b-b486-443bdea104e2" containerName="istio-proxy"
Oct 31 07:46:14.728149 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.728124    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:14.746517 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.746479    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"]
Oct 31 07:46:14.754345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.754272    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:46:14.754345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.754340    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:46:14.772746 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.772703    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bktk\" (UniqueName: \"kubernetes.io/projected/ae181a5d-0769-40e4-8875-e97f09d9996a-kube-api-access-4bktk\") pod \"maas-api-revoke-sub-6ac71db4-k924w\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:14.772995 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.772793    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac71db4-k924w\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:14.772995 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.772883    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac71db4-k924w\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:14.780350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.780321    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:46:14.780601 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.780577    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:46:14.803205 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.803166    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"]
Oct 31 07:46:14.803467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.803433    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy" containerID="cri-o://547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3" gracePeriod=30
Oct 31 07:46:14.823508 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.823472    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"]
Oct 31 07:46:14.823771 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.823742    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy" containerID="cri-o://276b1fb5db439d4297b4dd224cad6d237ef8077326a1beb3cbab9652cf55316b" gracePeriod=30
Oct 31 07:46:14.828391 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.828360    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"]
Oct 31 07:46:14.828672 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.828617    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy" containerID="cri-o://21e40bc21069aaa1fb280c7c64872677d9e46b547c6d94ce6e9ce30bb038f78e" gracePeriod=30
Oct 31 07:46:14.874099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.874058    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bktk\" (UniqueName: \"kubernetes.io/projected/ae181a5d-0769-40e4-8875-e97f09d9996a-kube-api-access-4bktk\") pod \"maas-api-revoke-sub-6ac71db4-k924w\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:14.874307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.874109    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac71db4-k924w\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:14.874307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.874146    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac71db4-k924w\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:14.874823 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.874800    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac71db4-k924w\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:14.876558 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.876537    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac71db4-k924w\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:14.894825 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:14.894791    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bktk\" (UniqueName: \"kubernetes.io/projected/ae181a5d-0769-40e4-8875-e97f09d9996a-kube-api-access-4bktk\") pod \"maas-api-revoke-sub-6ac71db4-k924w\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:15.042664 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:15.042571    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:15.352952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:15.352919    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"]
Oct 31 07:46:15.353672 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:46:15.353640    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae181a5d_0769_40e4_8875_e97f09d9996a.slice/crio-6d584fb36bba483423e623e2ea658644a86a208c77a6f09532ae1ab828f7ac77 WatchSource:0}: Error finding container 6d584fb36bba483423e623e2ea658644a86a208c77a6f09532ae1ab828f7ac77: Status 404 returned error can't find the container with id 6d584fb36bba483423e623e2ea658644a86a208c77a6f09532ae1ab828f7ac77
Oct 31 07:46:16.062602 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:16.062560    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w" event={"ID":"ae181a5d-0769-40e4-8875-e97f09d9996a","Type":"ContainerStarted","Data":"21e9f0dfc201ce739738f849faca17d6689a7b8b9d5bc89fb5ce65d8126dc5a4"}
Oct 31 07:46:16.063088 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:16.062623    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w" event={"ID":"ae181a5d-0769-40e4-8875-e97f09d9996a","Type":"ContainerStarted","Data":"6d584fb36bba483423e623e2ea658644a86a208c77a6f09532ae1ab828f7ac77"}
Oct 31 07:46:16.101941 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:16.101881    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w" podStartSLOduration=2.101866148 podStartE2EDuration="2.101866148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:46:16.100446828 +0000 UTC m=+1040.239607415" watchObservedRunningTime="2026-10-31 07:46:16.101866148 +0000 UTC m=+1040.241026735"
Oct 31 07:46:16.510081 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:46:16.510042    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:46:17.078730 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:17.078690    5829 generic.go:356] "Generic (PLEG): container finished" podID="ae181a5d-0769-40e4-8875-e97f09d9996a" containerID="21e9f0dfc201ce739738f849faca17d6689a7b8b9d5bc89fb5ce65d8126dc5a4" exitCode=0
Oct 31 07:46:17.079162 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:17.078774    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w" event={"ID":"ae181a5d-0769-40e4-8875-e97f09d9996a","Type":"ContainerDied","Data":"21e9f0dfc201ce739738f849faca17d6689a7b8b9d5bc89fb5ce65d8126dc5a4"}
Oct 31 07:46:18.229676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.229647    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:18.331455 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.331425    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae181a5d-0769-40e4-8875-e97f09d9996a-kube-api-access-4bktk\" (UniqueName: \"kubernetes.io/projected/ae181a5d-0769-40e4-8875-e97f09d9996a-kube-api-access-4bktk\") pod \"ae181a5d-0769-40e4-8875-e97f09d9996a\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") "
Oct 31 07:46:18.331625 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.331479    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-openshift-ca\") pod \"ae181a5d-0769-40e4-8875-e97f09d9996a\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") "
Oct 31 07:46:18.331625 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.331511    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-xks-ca\") pod \"ae181a5d-0769-40e4-8875-e97f09d9996a\" (UID: \"ae181a5d-0769-40e4-8875-e97f09d9996a\") "
Oct 31 07:46:18.331867 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.331840    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-openshift-ca" pod "ae181a5d-0769-40e4-8875-e97f09d9996a" (UID: "ae181a5d-0769-40e4-8875-e97f09d9996a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:18.331975 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.331960    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:18.333671 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.333637    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae181a5d-0769-40e4-8875-e97f09d9996a-kube-api-access-4bktk" pod "ae181a5d-0769-40e4-8875-e97f09d9996a" (UID: "ae181a5d-0769-40e4-8875-e97f09d9996a"). InnerVolumeSpecName "kube-api-access-4bktk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:18.333749 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.333670    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-xks-ca" pod "ae181a5d-0769-40e4-8875-e97f09d9996a" (UID: "ae181a5d-0769-40e4-8875-e97f09d9996a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:46:18.433447 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.433413    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bktk\" (UniqueName: \"kubernetes.io/projected/ae181a5d-0769-40e4-8875-e97f09d9996a-kube-api-access-4bktk\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:18.433447 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.433447    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae181a5d-0769-40e4-8875-e97f09d9996a-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:18.544605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:18.544562    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg" podUID="81d626ba-700d-40f2-9aee-4edc15c80b85" containerName="payload-processing" containerID="cri-o://38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172" gracePeriod=25
Oct 31 07:46:19.100025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:19.099994    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w" event={"ID":"ae181a5d-0769-40e4-8875-e97f09d9996a","Type":"ContainerDied","Data":"6d584fb36bba483423e623e2ea658644a86a208c77a6f09532ae1ab828f7ac77"}
Oct 31 07:46:19.100025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:19.100025    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"
Oct 31 07:46:19.100248 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:19.100047    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d584fb36bba483423e623e2ea658644a86a208c77a6f09532ae1ab828f7ac77"
Oct 31 07:46:20.089497 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.089472    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:46:20.111779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.111263    5829 generic.go:356] "Generic (PLEG): container finished" podID="a327099c-8eca-483f-b558-f36f1edb0d97" containerID="547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3" exitCode=0
Oct 31 07:46:20.111779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.111327    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" event={"ID":"a327099c-8eca-483f-b558-f36f1edb0d97","Type":"ContainerDied","Data":"547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3"}
Oct 31 07:46:20.111779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.111354    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"
Oct 31 07:46:20.111779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.111400    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z" event={"ID":"a327099c-8eca-483f-b558-f36f1edb0d97","Type":"ContainerDied","Data":"d8937ba19dc678b036fd0f85bd1a80dbb8cacecb14d2fd5fe55ed0e4ed7350eb"}
Oct 31 07:46:20.111779 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.111431    5829 scope.go:122] "RemoveContainer" containerID="547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3"
Oct 31 07:46:20.114009 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.113980    5829 generic.go:356] "Generic (PLEG): container finished" podID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerID="276b1fb5db439d4297b4dd224cad6d237ef8077326a1beb3cbab9652cf55316b" exitCode=0
Oct 31 07:46:20.114128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.114053    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" event={"ID":"d0974f3c-81c0-46f2-8500-bc7893afc3ea","Type":"ContainerDied","Data":"276b1fb5db439d4297b4dd224cad6d237ef8077326a1beb3cbab9652cf55316b"}
Oct 31 07:46:20.116338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.116311    5829 generic.go:356] "Generic (PLEG): container finished" podID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerID="21e40bc21069aaa1fb280c7c64872677d9e46b547c6d94ce6e9ce30bb038f78e" exitCode=0
Oct 31 07:46:20.116469 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.116312    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" event={"ID":"e227c5a8-109d-4570-a1bb-b15de0ac0cfd","Type":"ContainerDied","Data":"21e40bc21069aaa1fb280c7c64872677d9e46b547c6d94ce6e9ce30bb038f78e"}
Oct 31 07:46:20.124734 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.124710    5829 scope.go:122] "RemoveContainer" containerID="547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3"
Oct 31 07:46:20.125061 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:46:20.125040    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3\": container with ID starting with 547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3 not found: ID does not exist" containerID="547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3"
Oct 31 07:46:20.125155 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.125073    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3"} err="failed to get container status \"547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3\": rpc error: code = NotFound desc = could not find container \"547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3\": container with ID starting with 547f10e55fa5934c434d008c759fab33c7acd13337920d0d1e551c6689eb0ce3 not found: ID does not exist"
Oct 31 07:46:20.162017 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.161988    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:46:20.165403 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.165372    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:46:20.257424 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257380    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-credential-socket\") pod \"a327099c-8eca-483f-b558-f36f1edb0d97\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") "
Oct 31 07:46:20.257813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257456    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-data\") pod \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") "
Oct 31 07:46:20.257813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257519    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-socket\") pod \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") "
Oct 31 07:46:20.257813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257554    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-istio-token\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-istio-token\") pod \"a327099c-8eca-483f-b558-f36f1edb0d97\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") "
Oct 31 07:46:20.257813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257585    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-envoy\") pod \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") "
Oct 31 07:46:20.257813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257629    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a327099c-8eca-483f-b558-f36f1edb0d97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a327099c-8eca-483f-b558-f36f1edb0d97-istio-podinfo\") pod \"a327099c-8eca-483f-b558-f36f1edb0d97\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") "
Oct 31 07:46:20.257813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257662    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istiod-ca-cert\") pod \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") "
Oct 31 07:46:20.257813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257702    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-credential-socket\") pod \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") "
Oct 31 07:46:20.257813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257772    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a327099c-8eca-483f-b558-f36f1edb0d97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a327099c-8eca-483f-b558-f36f1edb0d97-istiod-ca-cert\") pod \"a327099c-8eca-483f-b558-f36f1edb0d97\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") "
Oct 31 07:46:20.257813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257807    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-token\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-token\") pod \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257841    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-kube-api-access-tljm2\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-kube-api-access-tljm2\") pod \"a327099c-8eca-483f-b558-f36f1edb0d97\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257883    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-certs\") pod \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257917    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-socket\") pod \"a327099c-8eca-483f-b558-f36f1edb0d97\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257954    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-podinfo\") pod \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.257990    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-credential-socket\") pod \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258020    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istiod-ca-cert\") pod \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258068    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-socket\") pod \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258108    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-certs\") pod \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258151    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-data\") pod \"a327099c-8eca-483f-b558-f36f1edb0d97\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258199    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-kube-api-access-td5wk\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-kube-api-access-td5wk\") pod \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258244    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-certs\") pod \"a327099c-8eca-483f-b558-f36f1edb0d97\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258335    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-envoy\") pod \"a327099c-8eca-483f-b558-f36f1edb0d97\" (UID: \"a327099c-8eca-483f-b558-f36f1edb0d97\") "
Oct 31 07:46:20.258381 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258368    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-kube-api-access-cscnl\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-kube-api-access-cscnl\") pod \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") "
Oct 31 07:46:20.259007 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258405    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-data\") pod \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") "
Oct 31 07:46:20.259007 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258472    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-token\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-token\") pod \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") "
Oct 31 07:46:20.259007 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258508    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-envoy\") pod \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\" (UID: \"d0974f3c-81c0-46f2-8500-bc7893afc3ea\") "
Oct 31 07:46:20.259007 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.258561    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-podinfo\") pod \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\" (UID: \"e227c5a8-109d-4570-a1bb-b15de0ac0cfd\") "
Oct 31 07:46:20.259740 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.259705    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-credential-socket" pod "a327099c-8eca-483f-b558-f36f1edb0d97" (UID: "a327099c-8eca-483f-b558-f36f1edb0d97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.259740 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.259724    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-socket" pod "e227c5a8-109d-4570-a1bb-b15de0ac0cfd" (UID: "e227c5a8-109d-4570-a1bb-b15de0ac0cfd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.259920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.259826    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istiod-ca-cert" pod "d0974f3c-81c0-46f2-8500-bc7893afc3ea" (UID: "d0974f3c-81c0-46f2-8500-bc7893afc3ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:20.259995 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.259972    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-data" pod "d0974f3c-81c0-46f2-8500-bc7893afc3ea" (UID: "d0974f3c-81c0-46f2-8500-bc7893afc3ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.260215 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.260185    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-certs" pod "a327099c-8eca-483f-b558-f36f1edb0d97" (UID: "a327099c-8eca-483f-b558-f36f1edb0d97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.260352 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.260254    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istiod-ca-cert" pod "e227c5a8-109d-4570-a1bb-b15de0ac0cfd" (UID: "e227c5a8-109d-4570-a1bb-b15de0ac0cfd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:20.260477 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.260454    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-socket" pod "d0974f3c-81c0-46f2-8500-bc7893afc3ea" (UID: "d0974f3c-81c0-46f2-8500-bc7893afc3ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.260565 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.260541    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-credential-socket" pod "e227c5a8-109d-4570-a1bb-b15de0ac0cfd" (UID: "e227c5a8-109d-4570-a1bb-b15de0ac0cfd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.260752 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.260727    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-certs" pod "d0974f3c-81c0-46f2-8500-bc7893afc3ea" (UID: "d0974f3c-81c0-46f2-8500-bc7893afc3ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.260891 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.260868    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a327099c-8eca-483f-b558-f36f1edb0d97-istiod-ca-cert" pod "a327099c-8eca-483f-b558-f36f1edb0d97" (UID: "a327099c-8eca-483f-b558-f36f1edb0d97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:20.260976 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.260955    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-data" pod "a327099c-8eca-483f-b558-f36f1edb0d97" (UID: "a327099c-8eca-483f-b558-f36f1edb0d97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.261979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.261779    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-credential-socket" pod "d0974f3c-81c0-46f2-8500-bc7893afc3ea" (UID: "d0974f3c-81c0-46f2-8500-bc7893afc3ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.261979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.261779    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-socket" pod "a327099c-8eca-483f-b558-f36f1edb0d97" (UID: "a327099c-8eca-483f-b558-f36f1edb0d97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.262554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.262157    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-istio-token" pod "a327099c-8eca-483f-b558-f36f1edb0d97" (UID: "a327099c-8eca-483f-b558-f36f1edb0d97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:20.262554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.262173    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-data" pod "e227c5a8-109d-4570-a1bb-b15de0ac0cfd" (UID: "e227c5a8-109d-4570-a1bb-b15de0ac0cfd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.263008 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.262734    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a327099c-8eca-483f-b558-f36f1edb0d97-istio-podinfo" pod "a327099c-8eca-483f-b558-f36f1edb0d97" (UID: "a327099c-8eca-483f-b558-f36f1edb0d97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:46:20.263008 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.262812    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-podinfo" pod "e227c5a8-109d-4570-a1bb-b15de0ac0cfd" (UID: "e227c5a8-109d-4570-a1bb-b15de0ac0cfd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:46:20.263008 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.262873    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-envoy" pod "e227c5a8-109d-4570-a1bb-b15de0ac0cfd" (UID: "e227c5a8-109d-4570-a1bb-b15de0ac0cfd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.263218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.263011    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-certs" pod "e227c5a8-109d-4570-a1bb-b15de0ac0cfd" (UID: "e227c5a8-109d-4570-a1bb-b15de0ac0cfd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.264546 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.264511    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-token" pod "d0974f3c-81c0-46f2-8500-bc7893afc3ea" (UID: "d0974f3c-81c0-46f2-8500-bc7893afc3ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:20.265141 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.265115    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-envoy" pod "d0974f3c-81c0-46f2-8500-bc7893afc3ea" (UID: "d0974f3c-81c0-46f2-8500-bc7893afc3ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.265643 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.265620    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-token" pod "e227c5a8-109d-4570-a1bb-b15de0ac0cfd" (UID: "e227c5a8-109d-4570-a1bb-b15de0ac0cfd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:20.266417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.266378    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-envoy" pod "a327099c-8eca-483f-b558-f36f1edb0d97" (UID: "a327099c-8eca-483f-b558-f36f1edb0d97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:20.266525 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.266425    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-podinfo" pod "d0974f3c-81c0-46f2-8500-bc7893afc3ea" (UID: "d0974f3c-81c0-46f2-8500-bc7893afc3ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:46:20.266592 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.266517    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-kube-api-access-tljm2" pod "a327099c-8eca-483f-b558-f36f1edb0d97" (UID: "a327099c-8eca-483f-b558-f36f1edb0d97"). InnerVolumeSpecName "kube-api-access-tljm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:20.266910 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.266883    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-kube-api-access-cscnl" pod "e227c5a8-109d-4570-a1bb-b15de0ac0cfd" (UID: "e227c5a8-109d-4570-a1bb-b15de0ac0cfd"). InnerVolumeSpecName "kube-api-access-cscnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:20.267005 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.266981    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-kube-api-access-td5wk" pod "d0974f3c-81c0-46f2-8500-bc7893afc3ea" (UID: "d0974f3c-81c0-46f2-8500-bc7893afc3ea"). InnerVolumeSpecName "kube-api-access-td5wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:20.360253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360173    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360201    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360213    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360222    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360231    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360242    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360252    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a327099c-8eca-483f-b558-f36f1edb0d97-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360260    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360268    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360276    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a327099c-8eca-483f-b558-f36f1edb0d97-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360319    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360332    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljm2\" (UniqueName: \"kubernetes.io/projected/a327099c-8eca-483f-b558-f36f1edb0d97-kube-api-access-tljm2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360345    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360357    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360370    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360380    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360388    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360397    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360404    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360413    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360421    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td5wk\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-kube-api-access-td5wk\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360429    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360438    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a327099c-8eca-483f-b558-f36f1edb0d97-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360446    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cscnl\" (UniqueName: \"kubernetes.io/projected/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-kube-api-access-cscnl\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360454    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e227c5a8-109d-4570-a1bb-b15de0ac0cfd-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360463    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.360669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.360471    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0974f3c-81c0-46f2-8500-bc7893afc3ea-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:20.461713 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.461679    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"]
Oct 31 07:46:20.465429 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:20.465400    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-cc44z"]
Oct 31 07:46:21.028601 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.028567    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"]
Oct 31 07:46:21.032221 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.032188    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a99c7c66-hbpp8"]
Oct 31 07:46:21.126344 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.126261    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"
Oct 31 07:46:21.126344 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.126277    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f" event={"ID":"d0974f3c-81c0-46f2-8500-bc7893afc3ea","Type":"ContainerDied","Data":"75f86d002ab9a19f68d43b823865eb157e8e1e92470f620e9077080b3898ee85"}
Oct 31 07:46:21.126871 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.126353    5829 scope.go:122] "RemoveContainer" containerID="276b1fb5db439d4297b4dd224cad6d237ef8077326a1beb3cbab9652cf55316b"
Oct 31 07:46:21.128054 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.128036    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"
Oct 31 07:46:21.128141 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.128055    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk" event={"ID":"e227c5a8-109d-4570-a1bb-b15de0ac0cfd","Type":"ContainerDied","Data":"6267e08edb1b7f960cfccd77b01f164cd9caa1fdefd9d6540900412179db4014"}
Oct 31 07:46:21.137357 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.137311    5829 scope.go:122] "RemoveContainer" containerID="21e40bc21069aaa1fb280c7c64872677d9e46b547c6d94ce6e9ce30bb038f78e"
Oct 31 07:46:21.185383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.185338    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"]
Oct 31 07:46:21.187795 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.187772    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-5pzbk"]
Oct 31 07:46:21.224519 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.224454    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"]
Oct 31 07:46:21.229406 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.229371    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4794f5-openshift-default-74dd558db5-xtk4f"]
Oct 31 07:46:21.973841 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.973802    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4434a386-3d84-41f3-929f-ec2960f03ba6" path="/var/lib/kubelet/pods/4434a386-3d84-41f3-929f-ec2960f03ba6/volumes"
Oct 31 07:46:21.974400 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.974379    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" path="/var/lib/kubelet/pods/a327099c-8eca-483f-b558-f36f1edb0d97/volumes"
Oct 31 07:46:21.974892 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.974872    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" path="/var/lib/kubelet/pods/d0974f3c-81c0-46f2-8500-bc7893afc3ea/volumes"
Oct 31 07:46:21.975691 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:21.975668    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" path="/var/lib/kubelet/pods/e227c5a8-109d-4570-a1bb-b15de0ac0cfd/volumes"
Oct 31 07:46:22.897151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:22.897109    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" containerID="cri-o://0961a92277fa1445be4d47c3917d46cbc7793ede11257f8fd4431f16a0faefe1" gracePeriod=45
Oct 31 07:46:23.003092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.003043    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused"
Oct 31 07:46:23.146568 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.146530    5829 generic.go:356] "Generic (PLEG): container finished" podID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerID="0961a92277fa1445be4d47c3917d46cbc7793ede11257f8fd4431f16a0faefe1" exitCode=0
Oct 31 07:46:23.146716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.146606    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" event={"ID":"91d1cb28-6264-4ac6-9a1b-5c19788c262c","Type":"ContainerDied","Data":"0961a92277fa1445be4d47c3917d46cbc7793ede11257f8fd4431f16a0faefe1"}
Oct 31 07:46:23.146716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.146663    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw" event={"ID":"91d1cb28-6264-4ac6-9a1b-5c19788c262c","Type":"ContainerDied","Data":"570481758b97439d1eb6c332c85fe7ceb0444fd80756cc52de7934846af716e0"}
Oct 31 07:46:23.146716 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.146678    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="570481758b97439d1eb6c332c85fe7ceb0444fd80756cc52de7934846af716e0"
Oct 31 07:46:23.151729 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.151677    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:46:23.293961 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.293921    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-dshm\") pod \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") "
Oct 31 07:46:23.294116 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294044    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-home\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-home\") pod \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") "
Oct 31 07:46:23.294116 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294086    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tls-certs\" (UniqueName: \"kubernetes.io/secret/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tls-certs\") pod \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") "
Oct 31 07:46:23.294208 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294151    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d1cb28-6264-4ac6-9a1b-5c19788c262c-kube-api-access-l6kq7\" (UniqueName: \"kubernetes.io/projected/91d1cb28-6264-4ac6-9a1b-5c19788c262c-kube-api-access-l6kq7\") pod \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") "
Oct 31 07:46:23.294208 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294191    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tmp-dir\") pod \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") "
Oct 31 07:46:23.294324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294228    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-model-cache\") pod \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\" (UID: \"91d1cb28-6264-4ac6-9a1b-5c19788c262c\") "
Oct 31 07:46:23.294396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294355    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-home" pod "91d1cb28-6264-4ac6-9a1b-5c19788c262c" (UID: "91d1cb28-6264-4ac6-9a1b-5c19788c262c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:23.294574 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294540    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-model-cache" pod "91d1cb28-6264-4ac6-9a1b-5c19788c262c" (UID: "91d1cb28-6264-4ac6-9a1b-5c19788c262c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:23.294687 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294567    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tmp-dir" pod "91d1cb28-6264-4ac6-9a1b-5c19788c262c" (UID: "91d1cb28-6264-4ac6-9a1b-5c19788c262c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:23.294793 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294768    5829 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-home\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:23.294793 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294794    5829 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tmp-dir\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:23.294986 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.294810    5829 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-model-cache\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:23.296366 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.296336    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tls-certs" pod "91d1cb28-6264-4ac6-9a1b-5c19788c262c" (UID: "91d1cb28-6264-4ac6-9a1b-5c19788c262c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:46:23.296448 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.296373    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d1cb28-6264-4ac6-9a1b-5c19788c262c-kube-api-access-l6kq7" pod "91d1cb28-6264-4ac6-9a1b-5c19788c262c" (UID: "91d1cb28-6264-4ac6-9a1b-5c19788c262c"). InnerVolumeSpecName "kube-api-access-l6kq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:23.296448 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.296399    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-dshm" pod "91d1cb28-6264-4ac6-9a1b-5c19788c262c" (UID: "91d1cb28-6264-4ac6-9a1b-5c19788c262c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:23.396414 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.396377    5829 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91d1cb28-6264-4ac6-9a1b-5c19788c262c-dshm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:23.396414 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.396405    5829 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91d1cb28-6264-4ac6-9a1b-5c19788c262c-tls-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:23.396414 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:23.396417    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6kq7\" (UniqueName: \"kubernetes.io/projected/91d1cb28-6264-4ac6-9a1b-5c19788c262c-kube-api-access-l6kq7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:24.153704 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:24.153674    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"
Oct 31 07:46:24.207797 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:24.207752    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"]
Oct 31 07:46:24.220061 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:24.220024    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-b4e8fd/simulator-w2-b4e8fd-kserve-858b66f9c4-9g9vw"]
Oct 31 07:46:25.974078 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:25.974041    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" path="/var/lib/kubelet/pods/91d1cb28-6264-4ac6-9a1b-5c19788c262c/volumes"
Oct 31 07:46:25.978893 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:25.978867    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg" podUID="81d626ba-700d-40f2-9aee-4edc15c80b85" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:46:27.034374 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:27.034339    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"]
Oct 31 07:46:27.047147 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:27.047113    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e990ba-pkb7p"]
Oct 31 07:46:27.398568 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:27.398538    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:46:27.978776 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:27.978726    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c236dbaa-6211-4c66-9057-eeb37250003f" path="/var/lib/kubelet/pods/c236dbaa-6211-4c66-9057-eeb37250003f/volumes"
Oct 31 07:46:29.050383 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:29.050346    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"]
Oct 31 07:46:29.072393 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:29.072357    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd02f45f-kltxs"]
Oct 31 07:46:29.074843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:29.074808    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"]
Oct 31 07:46:29.078732 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:29.078702    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518aac12-8p8tj"]
Oct 31 07:46:29.974522 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:29.974489    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53837a9b-2b09-467f-8a73-aaf5b437c068" path="/var/lib/kubelet/pods/53837a9b-2b09-467f-8a73-aaf5b437c068/volumes"
Oct 31 07:46:29.974947 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:29.974933    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbee989c-47ec-4629-b7f3-25394499b7f9" path="/var/lib/kubelet/pods/dbee989c-47ec-4629-b7f3-25394499b7f9/volumes"
Oct 31 07:46:31.027970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:31.027937    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"]
Oct 31 07:46:31.040579 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:31.040552    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b9bfe2-qq66k"]
Oct 31 07:46:32.002319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:32.002252    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e81a907-0cd8-4627-9caf-f8c126649d06" path="/var/lib/kubelet/pods/4e81a907-0cd8-4627-9caf-f8c126649d06/volumes"
Oct 31 07:46:32.029417 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:32.029381    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"]
Oct 31 07:46:32.042101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:32.042068    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ab5bda7-cw5g7"]
Oct 31 07:46:33.973345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:33.973309    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c75c9a5-adcb-4c71-bc3d-b91eddec7668" path="/var/lib/kubelet/pods/9c75c9a5-adcb-4c71-bc3d-b91eddec7668/volumes"
Oct 31 07:46:35.978364 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:35.978325    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg" podUID="81d626ba-700d-40f2-9aee-4edc15c80b85" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:46:36.032483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:36.032443    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"]
Oct 31 07:46:36.044984 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:36.044941    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8758b4b-ks5zs"]
Oct 31 07:46:36.978407 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:36.978371    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"]
Oct 31 07:46:36.978872 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:36.978601    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy" containerID="cri-o://c3cf3efab955a372d73a43a6562c5bb6f76de632f0a1e45fcbc6ae64cd64ded2" gracePeriod=30
Oct 31 07:46:36.993414 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:36.993375    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"]
Oct 31 07:46:36.993784 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:36.993725    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy" containerID="cri-o://05ae96899eee0995d9d499d4b34f7743f78a2672c5e707820b3eafa60357a7da" gracePeriod=30
Oct 31 07:46:37.000831 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:37.000801    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"]
Oct 31 07:46:37.001024 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:37.001003    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy" containerID="cri-o://8b70f4cd38f42e16e01263a4dbed47b785dbfdc4dd22acc141cef9beb3d5a840" gracePeriod=30
Oct 31 07:46:37.083020 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:37.082979    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:46:37.974060 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:37.974022    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4" path="/var/lib/kubelet/pods/52f4cd1c-7e7d-4ac3-8f5a-bf9b1615bdf4/volumes"
Oct 31 07:46:38.805455 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:38.805397    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:46:38.903024 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:38.902984    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d626ba-700d-40f2-9aee-4edc15c80b85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d626ba-700d-40f2-9aee-4edc15c80b85-plugins-config-volume\") pod \"81d626ba-700d-40f2-9aee-4edc15c80b85\" (UID: \"81d626ba-700d-40f2-9aee-4edc15c80b85\") "
Oct 31 07:46:38.903189 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:38.903074    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d626ba-700d-40f2-9aee-4edc15c80b85-kube-api-access-g8fxb\" (UniqueName: \"kubernetes.io/projected/81d626ba-700d-40f2-9aee-4edc15c80b85-kube-api-access-g8fxb\") pod \"81d626ba-700d-40f2-9aee-4edc15c80b85\" (UID: \"81d626ba-700d-40f2-9aee-4edc15c80b85\") "
Oct 31 07:46:38.903484 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:38.903455    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d626ba-700d-40f2-9aee-4edc15c80b85-plugins-config-volume" pod "81d626ba-700d-40f2-9aee-4edc15c80b85" (UID: "81d626ba-700d-40f2-9aee-4edc15c80b85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:38.905233 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:38.905208    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d626ba-700d-40f2-9aee-4edc15c80b85-kube-api-access-g8fxb" pod "81d626ba-700d-40f2-9aee-4edc15c80b85" (UID: "81d626ba-700d-40f2-9aee-4edc15c80b85"). InnerVolumeSpecName "kube-api-access-g8fxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:39.004729 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.004691    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81d626ba-700d-40f2-9aee-4edc15c80b85-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:39.004729 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.004725    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8fxb\" (UniqueName: \"kubernetes.io/projected/81d626ba-700d-40f2-9aee-4edc15c80b85-kube-api-access-g8fxb\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:39.277511 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.277475    5829 generic.go:356] "Generic (PLEG): container finished" podID="81d626ba-700d-40f2-9aee-4edc15c80b85" containerID="38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172" exitCode=0
Oct 31 07:46:39.277689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.277526    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"
Oct 31 07:46:39.277689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.277560    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg" event={"ID":"81d626ba-700d-40f2-9aee-4edc15c80b85","Type":"ContainerDied","Data":"38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172"}
Oct 31 07:46:39.277689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.277619    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg" event={"ID":"81d626ba-700d-40f2-9aee-4edc15c80b85","Type":"ContainerDied","Data":"797c235cc25ee9aaffd7feada84052b9c446398167aeb2867da392f11e080f0b"}
Oct 31 07:46:39.277689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.277648    5829 scope.go:122] "RemoveContainer" containerID="38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172"
Oct 31 07:46:39.288688 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.288665    5829 scope.go:122] "RemoveContainer" containerID="38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172"
Oct 31 07:46:39.288988 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:46:39.288964    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172\": container with ID starting with 38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172 not found: ID does not exist" containerID="38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172"
Oct 31 07:46:39.289039 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.288999    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172"} err="failed to get container status \"38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172\": rpc error: code = NotFound desc = could not find container \"38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172\": container with ID starting with 38f593be1b94ec5148a1dec93d0438d95769b013988d0934743a178010edb172 not found: ID does not exist"
Oct 31 07:46:39.333802 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.333772    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"]
Oct 31 07:46:39.346180 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.346151    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-b4e8fd-75b7c6774d-6sccg"]
Oct 31 07:46:39.973689 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:39.973654    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d626ba-700d-40f2-9aee-4edc15c80b85" path="/var/lib/kubelet/pods/81d626ba-700d-40f2-9aee-4edc15c80b85/volumes"
Oct 31 07:46:41.769160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:41.769119    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 07:46:41.769598 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:41.769189    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:46:41.891311 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:41.891246    5829 patch_prober.go:29] interesting pod/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 07:46:41.891501 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:41.891333    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 07:46:42.030259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.030222    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"]
Oct 31 07:46:42.040935 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.040905    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45190c6a-xpssh"]
Oct 31 07:46:42.351487 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.351230    5829 generic.go:356] "Generic (PLEG): container finished" podID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerID="05ae96899eee0995d9d499d4b34f7743f78a2672c5e707820b3eafa60357a7da" exitCode=0
Oct 31 07:46:42.351487 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.351374    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" event={"ID":"d7ab464c-8519-4d3d-9d97-10cade920d34","Type":"ContainerDied","Data":"05ae96899eee0995d9d499d4b34f7743f78a2672c5e707820b3eafa60357a7da"}
Oct 31 07:46:42.386186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.385711    5829 generic.go:356] "Generic (PLEG): container finished" podID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerID="8b70f4cd38f42e16e01263a4dbed47b785dbfdc4dd22acc141cef9beb3d5a840" exitCode=0
Oct 31 07:46:42.386186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.385942    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" event={"ID":"6925ac9a-0be2-4520-96fa-f4126b2fe58b","Type":"ContainerDied","Data":"8b70f4cd38f42e16e01263a4dbed47b785dbfdc4dd22acc141cef9beb3d5a840"}
Oct 31 07:46:42.389824 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.389637    5829 generic.go:356] "Generic (PLEG): container finished" podID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerID="c3cf3efab955a372d73a43a6562c5bb6f76de632f0a1e45fcbc6ae64cd64ded2" exitCode=0
Oct 31 07:46:42.389824 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.389747    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" event={"ID":"39acad66-7d35-49b2-8b03-8f900db5c8df","Type":"ContainerDied","Data":"c3cf3efab955a372d73a43a6562c5bb6f76de632f0a1e45fcbc6ae64cd64ded2"}
Oct 31 07:46:42.390691 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.390301    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:46:42.394382 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.394275    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:46:42.430883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.430854    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:46:42.443731 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.443691    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-kube-api-access-59gld\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-kube-api-access-59gld\") pod \"39acad66-7d35-49b2-8b03-8f900db5c8df\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") "
Oct 31 07:46:42.443898 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.443757    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-envoy\") pod \"39acad66-7d35-49b2-8b03-8f900db5c8df\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") "
Oct 31 07:46:42.443898 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.443877    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-credential-socket\") pod \"39acad66-7d35-49b2-8b03-8f900db5c8df\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") "
Oct 31 07:46:42.444040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.443918    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-podinfo\") pod \"39acad66-7d35-49b2-8b03-8f900db5c8df\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") "
Oct 31 07:46:42.444040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.443963    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39acad66-7d35-49b2-8b03-8f900db5c8df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acad66-7d35-49b2-8b03-8f900db5c8df-istiod-ca-cert\") pod \"39acad66-7d35-49b2-8b03-8f900db5c8df\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") "
Oct 31 07:46:42.444040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.443994    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-socket\") pod \"39acad66-7d35-49b2-8b03-8f900db5c8df\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") "
Oct 31 07:46:42.444209 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.444076    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-token\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-token\") pod \"39acad66-7d35-49b2-8b03-8f900db5c8df\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") "
Oct 31 07:46:42.444209 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.444131    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-certs\") pod \"39acad66-7d35-49b2-8b03-8f900db5c8df\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") "
Oct 31 07:46:42.444209 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.444185    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-data\") pod \"39acad66-7d35-49b2-8b03-8f900db5c8df\" (UID: \"39acad66-7d35-49b2-8b03-8f900db5c8df\") "
Oct 31 07:46:42.445578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.445548    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39acad66-7d35-49b2-8b03-8f900db5c8df-istiod-ca-cert" pod "39acad66-7d35-49b2-8b03-8f900db5c8df" (UID: "39acad66-7d35-49b2-8b03-8f900db5c8df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:42.446452 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.446426    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-data" pod "39acad66-7d35-49b2-8b03-8f900db5c8df" (UID: "39acad66-7d35-49b2-8b03-8f900db5c8df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.446870 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.446644    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-socket" pod "39acad66-7d35-49b2-8b03-8f900db5c8df" (UID: "39acad66-7d35-49b2-8b03-8f900db5c8df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.447272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.447250    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-certs" pod "39acad66-7d35-49b2-8b03-8f900db5c8df" (UID: "39acad66-7d35-49b2-8b03-8f900db5c8df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.449589 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.447514    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-credential-socket" pod "39acad66-7d35-49b2-8b03-8f900db5c8df" (UID: "39acad66-7d35-49b2-8b03-8f900db5c8df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.449808 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.449775    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-podinfo" pod "39acad66-7d35-49b2-8b03-8f900db5c8df" (UID: "39acad66-7d35-49b2-8b03-8f900db5c8df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:46:42.456701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.456615    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-kube-api-access-59gld" pod "39acad66-7d35-49b2-8b03-8f900db5c8df" (UID: "39acad66-7d35-49b2-8b03-8f900db5c8df"). InnerVolumeSpecName "kube-api-access-59gld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:42.457763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.457714    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-token" pod "39acad66-7d35-49b2-8b03-8f900db5c8df" (UID: "39acad66-7d35-49b2-8b03-8f900db5c8df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:42.459817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.459778    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-envoy" pod "39acad66-7d35-49b2-8b03-8f900db5c8df" (UID: "39acad66-7d35-49b2-8b03-8f900db5c8df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.545937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.545891    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-certs\") pod \"d7ab464c-8519-4d3d-9d97-10cade920d34\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") "
Oct 31 07:46:42.546145 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546025    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istiod-ca-cert\") pod \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") "
Oct 31 07:46:42.546145 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546061    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-envoy\") pod \"d7ab464c-8519-4d3d-9d97-10cade920d34\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") "
Oct 31 07:46:42.546145 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546119    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-token\") pod \"d7ab464c-8519-4d3d-9d97-10cade920d34\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") "
Oct 31 07:46:42.546307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546166    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-envoy\") pod \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") "
Oct 31 07:46:42.546307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546197    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-socket\") pod \"d7ab464c-8519-4d3d-9d97-10cade920d34\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") "
Oct 31 07:46:42.546307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546242    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-certs\") pod \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") "
Oct 31 07:46:42.546307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546304    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-podinfo\") pod \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") "
Oct 31 07:46:42.546442 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546370    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-credential-socket\") pod \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") "
Oct 31 07:46:42.546442 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546410    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-data\") pod \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") "
Oct 31 07:46:42.546513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546472    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-token\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-token\") pod \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") "
Oct 31 07:46:42.546545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546514    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-socket\") pod \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") "
Oct 31 07:46:42.546583 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546545    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-kube-api-access-dlxbj\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-kube-api-access-dlxbj\") pod \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\" (UID: \"6925ac9a-0be2-4520-96fa-f4126b2fe58b\") "
Oct 31 07:46:42.546618 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546583    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-credential-socket\") pod \"d7ab464c-8519-4d3d-9d97-10cade920d34\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") "
Oct 31 07:46:42.546659 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546643    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7ab464c-8519-4d3d-9d97-10cade920d34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab464c-8519-4d3d-9d97-10cade920d34-istiod-ca-cert\") pod \"d7ab464c-8519-4d3d-9d97-10cade920d34\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") "
Oct 31 07:46:42.546694 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546676    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-data\") pod \"d7ab464c-8519-4d3d-9d97-10cade920d34\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") "
Oct 31 07:46:42.546727 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546707    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-kube-api-access-cm2s8\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-kube-api-access-cm2s8\") pod \"d7ab464c-8519-4d3d-9d97-10cade920d34\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") "
Oct 31 07:46:42.546762 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.546743    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-podinfo\") pod \"d7ab464c-8519-4d3d-9d97-10cade920d34\" (UID: \"d7ab464c-8519-4d3d-9d97-10cade920d34\") "
Oct 31 07:46:42.547220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547202    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.547253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547230    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.547308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547249    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39acad66-7d35-49b2-8b03-8f900db5c8df-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.547308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547266    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.547308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547298    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.547406 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547314    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.547406 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547327    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.547406 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547340    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59gld\" (UniqueName: \"kubernetes.io/projected/39acad66-7d35-49b2-8b03-8f900db5c8df-kube-api-access-59gld\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.547406 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547353    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39acad66-7d35-49b2-8b03-8f900db5c8df-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.547802 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547771    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-credential-socket" pod "6925ac9a-0be2-4520-96fa-f4126b2fe58b" (UID: "6925ac9a-0be2-4520-96fa-f4126b2fe58b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.548066 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547777    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-socket" pod "d7ab464c-8519-4d3d-9d97-10cade920d34" (UID: "d7ab464c-8519-4d3d-9d97-10cade920d34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.549948 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.547785    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-certs" pod "6925ac9a-0be2-4520-96fa-f4126b2fe58b" (UID: "6925ac9a-0be2-4520-96fa-f4126b2fe58b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.550086 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.548495    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ab464c-8519-4d3d-9d97-10cade920d34-istiod-ca-cert" pod "d7ab464c-8519-4d3d-9d97-10cade920d34" (UID: "d7ab464c-8519-4d3d-9d97-10cade920d34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:42.550686 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.548752    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-credential-socket" pod "d7ab464c-8519-4d3d-9d97-10cade920d34" (UID: "d7ab464c-8519-4d3d-9d97-10cade920d34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.550838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.548799    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istiod-ca-cert" pod "6925ac9a-0be2-4520-96fa-f4126b2fe58b" (UID: "6925ac9a-0be2-4520-96fa-f4126b2fe58b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:46:42.550965 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.548946    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-data" pod "d7ab464c-8519-4d3d-9d97-10cade920d34" (UID: "d7ab464c-8519-4d3d-9d97-10cade920d34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.551085 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.549095    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-certs" pod "d7ab464c-8519-4d3d-9d97-10cade920d34" (UID: "d7ab464c-8519-4d3d-9d97-10cade920d34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.551193 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.549233    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-socket" pod "6925ac9a-0be2-4520-96fa-f4126b2fe58b" (UID: "6925ac9a-0be2-4520-96fa-f4126b2fe58b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.551319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.549908    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-data" pod "6925ac9a-0be2-4520-96fa-f4126b2fe58b" (UID: "6925ac9a-0be2-4520-96fa-f4126b2fe58b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.554676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.554596    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-token" pod "6925ac9a-0be2-4520-96fa-f4126b2fe58b" (UID: "6925ac9a-0be2-4520-96fa-f4126b2fe58b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:42.554676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.554636    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-token" pod "d7ab464c-8519-4d3d-9d97-10cade920d34" (UID: "d7ab464c-8519-4d3d-9d97-10cade920d34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:42.554860 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.554743    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-envoy" pod "6925ac9a-0be2-4520-96fa-f4126b2fe58b" (UID: "6925ac9a-0be2-4520-96fa-f4126b2fe58b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.554967 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.554939    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-podinfo" pod "d7ab464c-8519-4d3d-9d97-10cade920d34" (UID: "d7ab464c-8519-4d3d-9d97-10cade920d34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:46:42.555272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.555233    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-kube-api-access-dlxbj" pod "6925ac9a-0be2-4520-96fa-f4126b2fe58b" (UID: "6925ac9a-0be2-4520-96fa-f4126b2fe58b"). InnerVolumeSpecName "kube-api-access-dlxbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:42.555272 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.555257    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-podinfo" pod "6925ac9a-0be2-4520-96fa-f4126b2fe58b" (UID: "6925ac9a-0be2-4520-96fa-f4126b2fe58b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 07:46:42.556175 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.556149    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-envoy" pod "d7ab464c-8519-4d3d-9d97-10cade920d34" (UID: "d7ab464c-8519-4d3d-9d97-10cade920d34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:46:42.556817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.556791    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-kube-api-access-cm2s8" pod "d7ab464c-8519-4d3d-9d97-10cade920d34" (UID: "d7ab464c-8519-4d3d-9d97-10cade920d34"). InnerVolumeSpecName "kube-api-access-cm2s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:46:42.648126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648072    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648120    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648129    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648137    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648145    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648126 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648153    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648161    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648169    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648179    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648212    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648220    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648229    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6925ac9a-0be2-4520-96fa-f4126b2fe58b-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648237    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxbj\" (UniqueName: \"kubernetes.io/projected/6925ac9a-0be2-4520-96fa-f4126b2fe58b-kube-api-access-dlxbj\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648247    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648255    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7ab464c-8519-4d3d-9d97-10cade920d34-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648263    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648270    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm2s8\" (UniqueName: \"kubernetes.io/projected/d7ab464c-8519-4d3d-9d97-10cade920d34-kube-api-access-cm2s8\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:42.648680 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:42.648305    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7ab464c-8519-4d3d-9d97-10cade920d34-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:46:43.399669 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.399636    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"
Oct 31 07:46:43.400162 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.399665    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm" event={"ID":"d7ab464c-8519-4d3d-9d97-10cade920d34","Type":"ContainerDied","Data":"4f9b36643e8fa46132a67dffd262ab89fc6593d737dc1919d4c6321d9f68e376"}
Oct 31 07:46:43.400162 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.399738    5829 scope.go:122] "RemoveContainer" containerID="05ae96899eee0995d9d499d4b34f7743f78a2672c5e707820b3eafa60357a7da"
Oct 31 07:46:43.401465 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.401437    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h" event={"ID":"6925ac9a-0be2-4520-96fa-f4126b2fe58b","Type":"ContainerDied","Data":"595ca5462afb24705b040cde51f82584e382c25a3db9c7d3833794c76be5604a"}
Oct 31 07:46:43.401606 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.401444    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"
Oct 31 07:46:43.418940 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.418914    5829 scope.go:122] "RemoveContainer" containerID="8b70f4cd38f42e16e01263a4dbed47b785dbfdc4dd22acc141cef9beb3d5a840"
Oct 31 07:46:43.436530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.434658    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw" event={"ID":"39acad66-7d35-49b2-8b03-8f900db5c8df","Type":"ContainerDied","Data":"d32f41785f9d434655a9f2b0edd6355d58bb946500c99a6b1d2467d8e770d477"}
Oct 31 07:46:43.436530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.434785    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"
Oct 31 07:46:43.483985 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.483176    5829 scope.go:122] "RemoveContainer" containerID="c3cf3efab955a372d73a43a6562c5bb6f76de632f0a1e45fcbc6ae64cd64ded2"
Oct 31 07:46:43.527333 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.525893    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"]
Oct 31 07:46:43.587589 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.587532    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-9mccm"]
Oct 31 07:46:43.592581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.592558    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"]
Oct 31 07:46:43.595865 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.595839    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-4chdw"]
Oct 31 07:46:43.650617 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.650423    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"]
Oct 31 07:46:43.676152 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.676048    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-b4e8fd-openshift-default-55864b8599-hwh5h"]
Oct 31 07:46:43.983889 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.983798    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" path="/var/lib/kubelet/pods/39acad66-7d35-49b2-8b03-8f900db5c8df/volumes"
Oct 31 07:46:43.984251 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.984236    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" path="/var/lib/kubelet/pods/6925ac9a-0be2-4520-96fa-f4126b2fe58b/volumes"
Oct 31 07:46:43.984662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.984645    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93edee68-d792-4393-ace2-e2a716cb6d0a" path="/var/lib/kubelet/pods/93edee68-d792-4393-ace2-e2a716cb6d0a/volumes"
Oct 31 07:46:43.985391 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:43.985371    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" path="/var/lib/kubelet/pods/d7ab464c-8519-4d3d-9d97-10cade920d34/volumes"
Oct 31 07:46:46.511009 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:46:46.510970    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:46:51.028128 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:51.028092    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"]
Oct 31 07:46:51.041004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:51.040970    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-febff933-2q6pn"]
Oct 31 07:46:51.973467 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:51.973429    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7ae355-5f9f-42d8-962e-b398dd9a16cb" path="/var/lib/kubelet/pods/9a7ae355-5f9f-42d8-962e-b398dd9a16cb/volumes"
Oct 31 07:46:54.028058 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:54.028022    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"]
Oct 31 07:46:54.040725 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:54.040688    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f468be1-jhgkc"]
Oct 31 07:46:55.974881 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:55.974849    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97aadeef-b4a9-4159-ae79-076e95f95cb4" path="/var/lib/kubelet/pods/97aadeef-b4a9-4159-ae79-076e95f95cb4/volumes"
Oct 31 07:46:57.599920 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:57.599887    5829 scope.go:122] "RemoveContainer" containerID="cdaca30d5ecc5de2cc8b735e2ac34998f9a0ddfb8c92da424339ad72cc43118f"
Oct 31 07:46:57.613236 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:57.613211    5829 scope.go:122] "RemoveContainer" containerID="7d88d36b0c3af9e61447ee13488ba9730c71ae20d1a01305dae24a54fe2d754d"
Oct 31 07:46:57.622504 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:46:57.622482    5829 scope.go:122] "RemoveContainer" containerID="0961a92277fa1445be4d47c3917d46cbc7793ede11257f8fd4431f16a0faefe1"
Oct 31 07:47:02.026877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:02.026843    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"]
Oct 31 07:47:02.039437 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:02.039403    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5441a05e-k4kcb"]
Oct 31 07:47:03.973488 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:03.973451    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ce4241-c77c-4a22-8c61-6b6f736a734d" path="/var/lib/kubelet/pods/10ce4241-c77c-4a22-8c61-6b6f736a734d/volumes"
Oct 31 07:47:05.027718 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:05.027680    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"]
Oct 31 07:47:05.040455 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:05.040420    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce101d99-vzq94"]
Oct 31 07:47:05.974054 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:05.974015    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb06198-4ae1-4b31-a462-92e409a7fff1" path="/var/lib/kubelet/pods/8fb06198-4ae1-4b31-a462-92e409a7fff1/volumes"
Oct 31 07:47:15.028763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:15.028674    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"]
Oct 31 07:47:15.041378 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:15.041336    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d20bfd47-9dbnh"]
Oct 31 07:47:15.973044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:15.973000    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f9597b-20d1-4022-bd76-e9942fdd88ff" path="/var/lib/kubelet/pods/a7f9597b-20d1-4022-bd76-e9942fdd88ff/volumes"
Oct 31 07:47:16.512725 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:47:16.512681    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:47:25.035230 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:25.035193    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"]
Oct 31 07:47:25.055593 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:25.050504    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7e07fc7-lhc2b"]
Oct 31 07:47:25.973925 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:25.973891    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f0aaaa-0edc-4427-938c-cd401a65e0a5" path="/var/lib/kubelet/pods/79f0aaaa-0edc-4427-938c-cd401a65e0a5/volumes"
Oct 31 07:47:29.028296 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:29.028251    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"]
Oct 31 07:47:29.041014 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:29.040983    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0a3ab8e-ksl98"]
Oct 31 07:47:29.973339 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:29.973272    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad412fdc-281e-4558-88d7-56960f953b45" path="/var/lib/kubelet/pods/ad412fdc-281e-4558-88d7-56960f953b45/volumes"
Oct 31 07:47:38.029044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:38.029006    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"]
Oct 31 07:47:38.041492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:38.041458    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8880ccd8-grpwx"]
Oct 31 07:47:39.973887 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:39.973852    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d202dea-f0ad-4fe1-9b03-be291a9b121a" path="/var/lib/kubelet/pods/0d202dea-f0ad-4fe1-9b03-be291a9b121a/volumes"
Oct 31 07:47:44.030496 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:44.030455    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"]
Oct 31 07:47:44.042993 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:44.042961    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2482a204-nt5lp"]
Oct 31 07:47:45.973746 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:45.973707    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bdecec-8df0-4148-aa9c-43f269ef37f5" path="/var/lib/kubelet/pods/70bdecec-8df0-4148-aa9c-43f269ef37f5/volumes"
Oct 31 07:47:46.511068 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:47:46.511028    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:47:55.035326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:55.035292    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"]
Oct 31 07:47:55.048032 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:55.048000    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d95db595-2jc5b"]
Oct 31 07:47:55.973828 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:55.973797    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dceb19a-5ea9-4b23-9d4f-163c57e36e17" path="/var/lib/kubelet/pods/6dceb19a-5ea9-4b23-9d4f-163c57e36e17/volumes"
Oct 31 07:47:57.708482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.708450    5829 scope.go:122] "RemoveContainer" containerID="1b8e28897df68a546d0ce9c101e2b50cd08915d16296173cf36182a357b39ed5"
Oct 31 07:47:57.718845 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.718821    5829 scope.go:122] "RemoveContainer" containerID="f67292100cae2f8a17301fcf20e2158c965dee48dd1fd4f484055d7d7c0ea92f"
Oct 31 07:47:57.728504 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.728482    5829 scope.go:122] "RemoveContainer" containerID="732b1555131a99468d47bcd797f048df8144681724657ddae3174ce6bcbfb29b"
Oct 31 07:47:57.738322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.738275    5829 scope.go:122] "RemoveContainer" containerID="eda8c43cc60836a744e5e9c8fabb4d51f806ce46d881750323efb2ff178ce89b"
Oct 31 07:47:57.747588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.747567    5829 scope.go:122] "RemoveContainer" containerID="d0590e486ea3881a1f79bddd7160f398bb4f7c4960dead068fe1ccdca4d697e3"
Oct 31 07:47:57.756533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.756507    5829 scope.go:122] "RemoveContainer" containerID="cd0af04ea5bab4a3c81abc90de29209f3b61c0fdea799e5a5cd1ac07cf5dda4a"
Oct 31 07:47:57.765627 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.765606    5829 scope.go:122] "RemoveContainer" containerID="07573383f6254be7a50e3c05159392a1c9493a1d213f4edb159762091a286ed8"
Oct 31 07:47:57.775102 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.775079    5829 scope.go:122] "RemoveContainer" containerID="6ca517689cc6c558a83a9d722116d3b37ea210395d3ab063d0a509ef3ef6aec5"
Oct 31 07:47:57.783954 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.783923    5829 scope.go:122] "RemoveContainer" containerID="0b1f14cdabb9ff7844b7fb9e434d344dd48aff5d2cd32c46e4919d0aaca847bf"
Oct 31 07:47:57.792557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.792538    5829 scope.go:122] "RemoveContainer" containerID="881b9f5a003a3044fcce1503b214369176ccd76cf27d229b83a0418812047cbc"
Oct 31 07:47:57.801338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.801319    5829 scope.go:122] "RemoveContainer" containerID="cc2919452a057a5e3eb447b345bbf524d94030e748e74a5c48108f6b845de8e4"
Oct 31 07:47:57.812459 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.812384    5829 scope.go:122] "RemoveContainer" containerID="b521173c5a38ccb0bd77ac99ed743bbc5ca1b83d4a14345285ae550130a1adcf"
Oct 31 07:47:57.821560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:47:57.821540    5829 scope.go:122] "RemoveContainer" containerID="cbcda8bd8c54c82cba7197b1358f6405d832331ab8d9e053961d4ca196bf793e"
Oct 31 07:48:05.598129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:05.598095    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:48:08.341307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:08.340382    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:48:08.341307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:08.340456    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:48:16.510666 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:48:16.510623    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:48:22.581246 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:22.581208    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:48:26.754763 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:26.754730    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:48:28.054773 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:28.054728    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"]
Oct 31 07:48:28.078499 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:28.078446    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d700421-47mdl"]
Oct 31 07:48:29.972813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:29.972777    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5502cd79-b262-407f-8a2f-2517aa1e7cb7" path="/var/lib/kubelet/pods/5502cd79-b262-407f-8a2f-2517aa1e7cb7/volumes"
Oct 31 07:48:41.027079 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:41.026995    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"]
Oct 31 07:48:41.040025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:41.039993    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d48db09e-qqnwf"]
Oct 31 07:48:41.979889 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:41.979852    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff220bf-a427-4dbe-b053-bc59908a2a3e" path="/var/lib/kubelet/pods/aff220bf-a427-4dbe-b053-bc59908a2a3e/volumes"
Oct 31 07:48:42.583860 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:42.583815    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:48:43.102356 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:43.102320    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"]
Oct 31 07:48:43.114318 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:43.114275    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-976fb0cc-sc5lm"]
Oct 31 07:48:43.974218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:43.974184    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b2f342-3d23-42f2-a068-e42f57cf0366" path="/var/lib/kubelet/pods/54b2f342-3d23-42f2-a068-e42f57cf0366/volumes"
Oct 31 07:48:46.510358 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:48:46.510308    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:48:57.907362 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:57.907329    5829 scope.go:122] "RemoveContainer" containerID="29999e2749e1d18bfcd5520aeda06fd5a0f5a443a7dd5bc1ff073ee1a07573ec"
Oct 31 07:48:57.917061 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:57.917041    5829 scope.go:122] "RemoveContainer" containerID="72481dcbb84bcaee5774395716f713b7e613db51d6b507c13678eed5db9ff44f"
Oct 31 07:48:57.926097 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:57.926073    5829 scope.go:122] "RemoveContainer" containerID="321dfdff9b4afeb5303e41449b061125309ba85d49a22ff3b36f823801f1eb7b"
Oct 31 07:48:57.934874 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:57.934853    5829 scope.go:122] "RemoveContainer" containerID="bf71f314fbe5e873b6bb01ecfc770e2a754f41f261d61bde85e773c72b3ff7a8"
Oct 31 07:48:57.943980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:57.943954    5829 scope.go:122] "RemoveContainer" containerID="8f70fabf9bfd8b6faf0b7f6b89b3534b8bb0655f027de519b70ee71dab1d5f01"
Oct 31 07:48:57.952692 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:57.952671    5829 scope.go:122] "RemoveContainer" containerID="e9d7e972d1c7e1115bfbf5d76959d54fcb62756180512164270d23b7ba90a1ec"
Oct 31 07:48:57.961036 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:57.961010    5829 scope.go:122] "RemoveContainer" containerID="73c974236c392db18c5b7293a7518127705936ab42cd9c15f8e801fb82cea524"
Oct 31 07:48:57.972827 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:57.972742    5829 scope.go:122] "RemoveContainer" containerID="8e1ab5cca387dbbe84b289bd178f5a4c79c94b414c3b88ae6c239713a5de87ae"
Oct 31 07:48:58.070623 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:58.070598    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:48:58.071072 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:48:58.071051    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:49:00.027118 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:00.027080    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"]
Oct 31 07:49:00.040166 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:00.040126    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b91fa08f-8h6gg"]
Oct 31 07:49:01.974136 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:01.974101    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230e6b54-1f3b-427b-9e5d-bd998203856d" path="/var/lib/kubelet/pods/230e6b54-1f3b-427b-9e5d-bd998203856d/volumes"
Oct 31 07:49:02.026565 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:02.026532    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"]
Oct 31 07:49:02.038981 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:02.038950    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b69cdb34-2zbbp"]
Oct 31 07:49:03.972718 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:03.972681    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c39ec502-6388-452a-a67a-b1f7be7d0319" path="/var/lib/kubelet/pods/c39ec502-6388-452a-a67a-b1f7be7d0319/volumes"
Oct 31 07:49:16.511539 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:49:16.511496    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:49:18.157575 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:18.157535    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:49:27.033968 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:27.033931    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"]
Oct 31 07:49:27.047405 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:27.047372    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"]
Oct 31 07:49:27.049535 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:27.049514    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117782a3-bv6xl"]
Oct 31 07:49:27.051374 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:27.051351    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc19f141-6kzlc"]
Oct 31 07:49:27.973700 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:27.973665    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d208261-0f6b-4417-8b6f-3058381ffb7f" path="/var/lib/kubelet/pods/9d208261-0f6b-4417-8b6f-3058381ffb7f/volumes"
Oct 31 07:49:27.974156 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:27.974142    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3878be-78b1-4621-ba23-1b5b4cfb7a72" path="/var/lib/kubelet/pods/9e3878be-78b1-4621-ba23-1b5b4cfb7a72/volumes"
Oct 31 07:49:31.027144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:31.027109    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"]
Oct 31 07:49:31.039656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:31.039622    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c158eef-9j9pc"]
Oct 31 07:49:31.976848 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:31.976804    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db35bfba-1b72-4636-b79d-8985b3070ffe" path="/var/lib/kubelet/pods/db35bfba-1b72-4636-b79d-8985b3070ffe/volumes"
Oct 31 07:49:35.026304 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:35.026243    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"]
Oct 31 07:49:35.039234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:35.039184    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cac1558a-f5r5q"]
Oct 31 07:49:35.973810 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:35.973776    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63062387-4639-49f0-a3f7-6b6150b199ce" path="/var/lib/kubelet/pods/63062387-4639-49f0-a3f7-6b6150b199ce/volumes"
Oct 31 07:49:41.026717 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:41.026678    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"]
Oct 31 07:49:41.040212 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:41.040178    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f7bf75b-nwmh2"]
Oct 31 07:49:41.042290 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:41.042256    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"]
Oct 31 07:49:41.044248 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:41.044227    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f09760dd-7ltcl"]
Oct 31 07:49:41.448671 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:41.448640    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:49:41.977045 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:41.977004    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="330b31f3-9f9f-4e09-8c5e-fa542b87c526" path="/var/lib/kubelet/pods/330b31f3-9f9f-4e09-8c5e-fa542b87c526/volumes"
Oct 31 07:49:41.977723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:41.977698    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d95c9d-2384-4cba-a30c-4823af7b31d1" path="/var/lib/kubelet/pods/98d95c9d-2384-4cba-a30c-4823af7b31d1/volumes"
Oct 31 07:49:45.024217 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:45.024181    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"]
Oct 31 07:49:45.037450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:45.037419    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2b0f715-kqgr6"]
Oct 31 07:49:45.974240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:45.974202    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b43209-4f04-4826-9f20-a86825f26d63" path="/var/lib/kubelet/pods/42b43209-4f04-4826-9f20-a86825f26d63/volumes"
Oct 31 07:49:46.510461 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:49:46.510416    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:49:51.779406 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:51.779369    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:49:52.028706 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:52.028669    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"]
Oct 31 07:49:52.041511 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:52.041436    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b4919ba-xjbhz"]
Oct 31 07:49:53.973514 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:53.973476    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a1f555-afc5-4705-a149-baef955742e6" path="/var/lib/kubelet/pods/72a1f555-afc5-4705-a149-baef955742e6/volumes"
Oct 31 07:49:56.190720 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:56.190672    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:49:58.035532 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:58.035494    5829 scope.go:122] "RemoveContainer" containerID="9c8c42737f9abeef72cc5cfd0d2ce07d7d41a09f2d5aec0584a4a2488792017d"
Oct 31 07:49:58.080244 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:58.080213    5829 scope.go:122] "RemoveContainer" containerID="e074f211f3d6303511151e82545f135b02224f91a1fae826369f645cb3c6978d"
Oct 31 07:49:58.091051 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:58.091026    5829 scope.go:122] "RemoveContainer" containerID="1efb939e9cb75b342959711fa55dab170332c11a4e08eabe86f7e65aeb66a979"
Oct 31 07:49:58.099912 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:49:58.099891    5829 scope.go:122] "RemoveContainer" containerID="24794dc39f8123f060150c51a529499062fdccd67572d47a42f9687733565972"
Oct 31 07:50:03.024531 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:03.024492    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"]
Oct 31 07:50:03.037184 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:03.037144    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-980d26c0-7j78w"]
Oct 31 07:50:03.973410 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:03.973376    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75a8cf6-1560-4982-9c99-6d59f996512b" path="/var/lib/kubelet/pods/b75a8cf6-1560-4982-9c99-6d59f996512b/volumes"
Oct 31 07:50:05.025816 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:05.025782    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"]
Oct 31 07:50:05.039003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:05.038964    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e93e4474-c6l5h"]
Oct 31 07:50:05.973516 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:05.973479    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6519d7-c1dc-40e4-b83a-a8a07670662c" path="/var/lib/kubelet/pods/0a6519d7-c1dc-40e4-b83a-a8a07670662c/volumes"
Oct 31 07:50:13.541833 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:13.541735    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:50:15.026592 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:15.026549    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"]
Oct 31 07:50:15.039226 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:15.039194    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ffdd5d6-wwz4g"]
Oct 31 07:50:15.976240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:15.976202    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17a2d17-acad-40a8-8c19-b1c641832ed9" path="/var/lib/kubelet/pods/b17a2d17-acad-40a8-8c19-b1c641832ed9/volumes"
Oct 31 07:50:16.515701 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:50:16.515657    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:50:18.684019 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:18.683981    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:50:19.027508 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:19.027423    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"]
Oct 31 07:50:19.040584 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:19.040548    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7dd675ee-v2q49"]
Oct 31 07:50:19.974019 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:19.973985    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d035bbe-d467-4f01-8325-8ccac7e65551" path="/var/lib/kubelet/pods/8d035bbe-d467-4f01-8325-8ccac7e65551/volumes"
Oct 31 07:50:32.025641 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:32.025602    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"]
Oct 31 07:50:32.038099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:32.038062    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3687bccb-9rqzs"]
Oct 31 07:50:33.028188 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:33.028150    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"]
Oct 31 07:50:33.039243 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:33.039210    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3061197b-7fwnq"]
Oct 31 07:50:33.973649 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:33.973609    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c52732f-cee9-421a-ad27-422e304db24e" path="/var/lib/kubelet/pods/5c52732f-cee9-421a-ad27-422e304db24e/volumes"
Oct 31 07:50:33.974033 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:33.974019    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874be886-01ff-4e96-be4f-fbe8c0ae7334" path="/var/lib/kubelet/pods/874be886-01ff-4e96-be4f-fbe8c0ae7334/volumes"
Oct 31 07:50:43.031596 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:43.031559    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"]
Oct 31 07:50:43.044421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:43.044392    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1f09f4c-wwqt4"]
Oct 31 07:50:43.973655 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:43.973619    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9a29cc-5c61-4200-b808-d95bc47fce1e" path="/var/lib/kubelet/pods/0f9a29cc-5c61-4200-b808-d95bc47fce1e/volumes"
Oct 31 07:50:44.784045 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:44.784009    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:50:46.511356 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:50:46.511321    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:50:49.593530 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:49.593497    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:50:58.228342 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.228310    5829 scope.go:122] "RemoveContainer" containerID="9fc40a611c69704a8cb1d9570c1ede3ab331a9d0db0fe81a2d6d730220a2143c"
Oct 31 07:50:58.237878 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.237859    5829 scope.go:122] "RemoveContainer" containerID="875a969d99ece404246aa14bae80252eaebc3c8119fc467035f32ac407b4e575"
Oct 31 07:50:58.246466 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.246435    5829 scope.go:122] "RemoveContainer" containerID="07bac55930f323462971c97140aedfc0013c55f8b7a04463064bcbc5827f37e4"
Oct 31 07:50:58.255507 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.255477    5829 scope.go:122] "RemoveContainer" containerID="e3884fc42ccbf918be57783c50c7de5aec6b6048e425886398e6295292b8f21a"
Oct 31 07:50:58.264433 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.264412    5829 scope.go:122] "RemoveContainer" containerID="3451b4a9f7f1e89209d511822f52de7ed5a2d26bf9ddc9a6529c1f98e10c5c6b"
Oct 31 07:50:58.273707 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.273687    5829 scope.go:122] "RemoveContainer" containerID="24955d0eaa9dc57a5670058e6c43f226fd4bd50ce9713d78d54364307bbf82fd"
Oct 31 07:50:58.282450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.282427    5829 scope.go:122] "RemoveContainer" containerID="eac814eefe9771feee095fd46150c57e9ad4cf782745ae1ff6a734c21d49429f"
Oct 31 07:50:58.291051 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.291029    5829 scope.go:122] "RemoveContainer" containerID="798785908f6bac78b85067be2c7efc657cb0772092febe5cd39a8fdd1dc73c4f"
Oct 31 07:50:58.299850 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.299829    5829 scope.go:122] "RemoveContainer" containerID="57a7face39c027f3a5654ffbad3a86581444deb0169ef2b7022259155372d54c"
Oct 31 07:50:58.308772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.308751    5829 scope.go:122] "RemoveContainer" containerID="8ed661cc63c7c671372f0f819bf7d5c6651856bdcdf40e6ab8c745c31e000b3d"
Oct 31 07:50:58.317662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.317642    5829 scope.go:122] "RemoveContainer" containerID="92e9636f6fade172d695b32ffbcb6f7fff3d33f19df4979c01c2b6a78bce757d"
Oct 31 07:50:58.327202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.327176    5829 scope.go:122] "RemoveContainer" containerID="873bf6b49e1edcc9391243e38826ae51f1afcaea6fabe98d19e275fd30f51e16"
Oct 31 07:50:58.336811 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.336788    5829 scope.go:122] "RemoveContainer" containerID="3892c17ccc70947282f99a790df8eb7ac7c2d037dc8af36483abd0a328bf86a6"
Oct 31 07:50:58.345508 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:50:58.345484    5829 scope.go:122] "RemoveContainer" containerID="26424ce028309f1637ae62ba161cad2e7c9b3362b3a5b84f2da3365cdbabf004"
Oct 31 07:51:04.847712 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:04.847681    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:51:07.946537 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:07.946487    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"]
Oct 31 07:51:07.947048 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:07.946834    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928" podUID="7f012873-83b7-487e-8da9-04fa08087316" containerName="discovery" containerID="cri-o://f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972" gracePeriod=30
Oct 31 07:51:08.246307 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.239387    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:51:08.369510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.367815    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-istio-token\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-istio-token\") pod \"7f012873-83b7-487e-8da9-04fa08087316\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") "
Oct 31 07:51:08.369510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.367877    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-cacerts\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-cacerts\") pod \"7f012873-83b7-487e-8da9-04fa08087316\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") "
Oct 31 07:51:08.369510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.367938    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-ca-configmap\") pod \"7f012873-83b7-487e-8da9-04fa08087316\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") "
Oct 31 07:51:08.369510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.368019    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-dns-cert\") pod \"7f012873-83b7-487e-8da9-04fa08087316\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") "
Oct 31 07:51:08.369510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.368156    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-kube-api-access-wsk97\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-kube-api-access-wsk97\") pod \"7f012873-83b7-487e-8da9-04fa08087316\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") "
Oct 31 07:51:08.369510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.368201    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f012873-83b7-487e-8da9-04fa08087316-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f012873-83b7-487e-8da9-04fa08087316-local-certs\") pod \"7f012873-83b7-487e-8da9-04fa08087316\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") "
Oct 31 07:51:08.369510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.368252    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-kubeconfig\") pod \"7f012873-83b7-487e-8da9-04fa08087316\" (UID: \"7f012873-83b7-487e-8da9-04fa08087316\") "
Oct 31 07:51:08.388269 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.385893    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-kubeconfig" pod "7f012873-83b7-487e-8da9-04fa08087316" (UID: "7f012873-83b7-487e-8da9-04fa08087316"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:51:08.388269 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.385924    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-ca-configmap" pod "7f012873-83b7-487e-8da9-04fa08087316" (UID: "7f012873-83b7-487e-8da9-04fa08087316"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:51:08.396679 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.396644    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f012873-83b7-487e-8da9-04fa08087316-local-certs" pod "7f012873-83b7-487e-8da9-04fa08087316" (UID: "7f012873-83b7-487e-8da9-04fa08087316"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 07:51:08.397052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.397028    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-kube-api-access-wsk97" pod "7f012873-83b7-487e-8da9-04fa08087316" (UID: "7f012873-83b7-487e-8da9-04fa08087316"). InnerVolumeSpecName "kube-api-access-wsk97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:51:08.397153 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.397065    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-cacerts" pod "7f012873-83b7-487e-8da9-04fa08087316" (UID: "7f012873-83b7-487e-8da9-04fa08087316"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:51:08.397153 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.397094    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-istio-token" pod "7f012873-83b7-487e-8da9-04fa08087316" (UID: "7f012873-83b7-487e-8da9-04fa08087316"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:51:08.398409 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.398363    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-dns-cert" pod "7f012873-83b7-487e-8da9-04fa08087316" (UID: "7f012873-83b7-487e-8da9-04fa08087316"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:51:08.473369 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.473211    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsk97\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-kube-api-access-wsk97\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:08.477313 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.473591    5829 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f012873-83b7-487e-8da9-04fa08087316-local-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:08.480595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.473628    5829 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-kubeconfig\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:08.480595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.480447    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f012873-83b7-487e-8da9-04fa08087316-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:08.480595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.480467    5829 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-cacerts\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:08.480595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.480484    5829 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-ca-configmap\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:08.480595 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.480499    5829 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7f012873-83b7-487e-8da9-04fa08087316-istio-csr-dns-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:08.530324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.503314    5829 generic.go:356] "Generic (PLEG): container finished" podID="7f012873-83b7-487e-8da9-04fa08087316" containerID="f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972" exitCode=0
Oct 31 07:51:08.530324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.503370    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"
Oct 31 07:51:08.530324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.503400    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928" event={"ID":"7f012873-83b7-487e-8da9-04fa08087316","Type":"ContainerDied","Data":"f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972"}
Oct 31 07:51:08.530324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.503450    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928" event={"ID":"7f012873-83b7-487e-8da9-04fa08087316","Type":"ContainerDied","Data":"9c4fb309fd722cb89a1e2ae5be13e461bdb57a2da917c1b729334f41b3790d97"}
Oct 31 07:51:08.530324 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.503478    5829 scope.go:122] "RemoveContainer" containerID="f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972"
Oct 31 07:51:08.590268 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.590241    5829 scope.go:122] "RemoveContainer" containerID="f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972"
Oct 31 07:51:08.590656 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:51:08.590635    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972\": container with ID starting with f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972 not found: ID does not exist" containerID="f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972"
Oct 31 07:51:08.590722 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.590667    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972"} err="failed to get container status \"f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972\": rpc error: code = NotFound desc = could not find container \"f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972\": container with ID starting with f9f243307a5f3ff4720ce40b3b45c65d7d16c4645509cdb014dfd468fd5a0972 not found: ID does not exist"
Oct 31 07:51:08.629314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.628875    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"]
Oct 31 07:51:08.656308 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:08.653541    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g8928"]
Oct 31 07:51:09.966068 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:09.966026    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:51:09.973262 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:09.973229    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f012873-83b7-487e-8da9-04fa08087316" path="/var/lib/kubelet/pods/7f012873-83b7-487e-8da9-04fa08087316/volumes"
Oct 31 07:51:13.705431 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.705394    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"]
Oct 31 07:51:13.705909 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.705894    5829 cpu_manager.go:414] "containerMap: removing container" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy"
Oct 31 07:51:13.705988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.705910    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy"
Oct 31 07:51:13.705988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.705944    5829 cpu_manager.go:414] "containerMap: removing container" podUID="81d626ba-700d-40f2-9aee-4edc15c80b85" containerName="payload-processing"
Oct 31 07:51:13.705988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.705952    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d626ba-700d-40f2-9aee-4edc15c80b85" containerName="payload-processing"
Oct 31 07:51:13.705988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.705967    5829 cpu_manager.go:414] "containerMap: removing container" podUID="ae181a5d-0769-40e4-8875-e97f09d9996a" containerName="revoke-keys"
Oct 31 07:51:13.705988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.705975    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae181a5d-0769-40e4-8875-e97f09d9996a" containerName="revoke-keys"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706001    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706008    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706016    5829 cpu_manager.go:414] "containerMap: removing container" podUID="7f012873-83b7-487e-8da9-04fa08087316" containerName="discovery"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706022    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f012873-83b7-487e-8da9-04fa08087316" containerName="discovery"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706031    5829 cpu_manager.go:414] "containerMap: removing container" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706035    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706040    5829 cpu_manager.go:414] "containerMap: removing container" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706045    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706054    5829 cpu_manager.go:414] "containerMap: removing container" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706058    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706066    5829 cpu_manager.go:414] "containerMap: removing container" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706070    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706082    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706088    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706166    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d626ba-700d-40f2-9aee-4edc15c80b85" containerName="payload-processing"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706175    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae181a5d-0769-40e4-8875-e97f09d9996a" containerName="revoke-keys"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706188    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7ab464c-8519-4d3d-9d97-10cade920d34" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706197    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f012873-83b7-487e-8da9-04fa08087316" containerName="discovery"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706206    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="a327099c-8eca-483f-b558-f36f1edb0d97" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706215    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="39acad66-7d35-49b2-8b03-8f900db5c8df" containerName="istio-proxy"
Oct 31 07:51:13.706218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706225    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="6925ac9a-0be2-4520-96fa-f4126b2fe58b" containerName="istio-proxy"
Oct 31 07:51:13.706997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706234    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d1cb28-6264-4ac6-9a1b-5c19788c262c" containerName="main"
Oct 31 07:51:13.706997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706245    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0974f3c-81c0-46f2-8500-bc7893afc3ea" containerName="istio-proxy"
Oct 31 07:51:13.706997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.706256    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="e227c5a8-109d-4570-a1bb-b15de0ac0cfd" containerName="istio-proxy"
Oct 31 07:51:13.711161 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.711137    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:13.728083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.728050    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:51:13.741073 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.741029    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:51:13.742897 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.742868    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"]
Oct 31 07:51:13.813101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.813062    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:51:13.813276 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.813179    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:51:13.839661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.839625    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whq8s\" (UniqueName: \"kubernetes.io/projected/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-kube-api-access-whq8s\") pod \"maas-api-revoke-sub-0b7e8edb-7chr9\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:13.839661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.839667    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7chr9\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:13.839863 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.839765    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7chr9\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:13.941296 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.941237    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whq8s\" (UniqueName: \"kubernetes.io/projected/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-kube-api-access-whq8s\") pod \"maas-api-revoke-sub-0b7e8edb-7chr9\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:13.941296 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.941298    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7chr9\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:13.941539 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.941352    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7chr9\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:13.941936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.941914    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7chr9\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:13.943710 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.943692    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7chr9\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:13.961083 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:13.961012    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whq8s\" (UniqueName: \"kubernetes.io/projected/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-kube-api-access-whq8s\") pod \"maas-api-revoke-sub-0b7e8edb-7chr9\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:14.028772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:14.028734    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:14.325938 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:14.325714    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"]
Oct 31 07:51:14.328450 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:51:14.328421    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8050eec0_fbfe_4df1_bf56_9fc0f4c933f3.slice/crio-b987a9510642cf84a64fb5bad3d2240761d5246b4d107e004e6bf193fdb4c8af WatchSource:0}: Error finding container b987a9510642cf84a64fb5bad3d2240761d5246b4d107e004e6bf193fdb4c8af: Status 404 returned error can't find the container with id b987a9510642cf84a64fb5bad3d2240761d5246b4d107e004e6bf193fdb4c8af
Oct 31 07:51:14.569974 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:14.569937    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9" event={"ID":"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3","Type":"ContainerStarted","Data":"b987a9510642cf84a64fb5bad3d2240761d5246b4d107e004e6bf193fdb4c8af"}
Oct 31 07:51:15.576554 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:15.576514    5829 generic.go:356] "Generic (PLEG): container finished" podID="8050eec0-fbfe-4df1-bf56-9fc0f4c933f3" containerID="89a14041a407c6817dccef9f258df5b83a347eb084b297b7cd754a4646b49a70" exitCode=0
Oct 31 07:51:15.576926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:15.576581    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9" event={"ID":"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3","Type":"ContainerDied","Data":"89a14041a407c6817dccef9f258df5b83a347eb084b297b7cd754a4646b49a70"}
Oct 31 07:51:16.510416 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:51:16.510377    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:51:16.725557 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.725531    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:16.772574 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.772494    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-kube-api-access-whq8s\" (UniqueName: \"kubernetes.io/projected/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-kube-api-access-whq8s\") pod \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") "
Oct 31 07:51:16.772747 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.772584    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-xks-ca\") pod \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") "
Oct 31 07:51:16.772747 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.772688    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-openshift-ca\") pod \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\" (UID: \"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3\") "
Oct 31 07:51:16.773091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.773051    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-openshift-ca" pod "8050eec0-fbfe-4df1-bf56-9fc0f4c933f3" (UID: "8050eec0-fbfe-4df1-bf56-9fc0f4c933f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:51:16.774668 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.774636    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-xks-ca" pod "8050eec0-fbfe-4df1-bf56-9fc0f4c933f3" (UID: "8050eec0-fbfe-4df1-bf56-9fc0f4c933f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:51:16.774781 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.774707    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-kube-api-access-whq8s" pod "8050eec0-fbfe-4df1-bf56-9fc0f4c933f3" (UID: "8050eec0-fbfe-4df1-bf56-9fc0f4c933f3"). InnerVolumeSpecName "kube-api-access-whq8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:51:16.874563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.874518    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:16.874563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.874559    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:16.874563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:16.874571    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whq8s\" (UniqueName: \"kubernetes.io/projected/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3-kube-api-access-whq8s\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:17.588821 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:17.588792    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"
Oct 31 07:51:17.588995 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:17.588818    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9" event={"ID":"8050eec0-fbfe-4df1-bf56-9fc0f4c933f3","Type":"ContainerDied","Data":"b987a9510642cf84a64fb5bad3d2240761d5246b4d107e004e6bf193fdb4c8af"}
Oct 31 07:51:17.588995 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:17.588865    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b987a9510642cf84a64fb5bad3d2240761d5246b4d107e004e6bf193fdb4c8af"
Oct 31 07:51:19.025719 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:19.025682    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"]
Oct 31 07:51:19.038537 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:19.038505    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac71db4-k924w"]
Oct 31 07:51:19.972971 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:19.972935    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae181a5d-0769-40e4-8875-e97f09d9996a" path="/var/lib/kubelet/pods/ae181a5d-0769-40e4-8875-e97f09d9996a/volumes"
Oct 31 07:51:24.434823 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.434787    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"]
Oct 31 07:51:24.435254 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.435239    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8050eec0-fbfe-4df1-bf56-9fc0f4c933f3" containerName="revoke-keys"
Oct 31 07:51:24.435336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.435254    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8050eec0-fbfe-4df1-bf56-9fc0f4c933f3" containerName="revoke-keys"
Oct 31 07:51:24.435377 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.435350    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8050eec0-fbfe-4df1-bf56-9fc0f4c933f3" containerName="revoke-keys"
Oct 31 07:51:24.438785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.438766    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.457720 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.457690    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"]
Oct 31 07:51:24.468371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.468343    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:51:24.468510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.468436    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:51:24.468510 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.468478    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:51:24.468597 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.468577    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:51:24.552160 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.552119    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5g999\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.552388 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.552176    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5g999\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.552388 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.552352    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5bjq\" (UniqueName: \"kubernetes.io/projected/b11e7548-483e-4939-ab37-f8e0410d7850-kube-api-access-l5bjq\") pod \"maas-api-revoke-sub-3d4156f1-5g999\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.653946 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.653907    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5g999\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.654142 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.653959    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5g999\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.654142 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.654003    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5bjq\" (UniqueName: \"kubernetes.io/projected/b11e7548-483e-4939-ab37-f8e0410d7850-kube-api-access-l5bjq\") pod \"maas-api-revoke-sub-3d4156f1-5g999\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.654723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.654694    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5g999\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.656339 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.656316    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5g999\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.674585 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.674554    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5bjq\" (UniqueName: \"kubernetes.io/projected/b11e7548-483e-4939-ab37-f8e0410d7850-kube-api-access-l5bjq\") pod \"maas-api-revoke-sub-3d4156f1-5g999\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.751325 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.751202    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:24.941600 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:24.941564    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:51:25.051305 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:25.051249    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"]
Oct 31 07:51:25.053856 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:51:25.053823    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb11e7548_483e_4939_ab37_f8e0410d7850.slice/crio-2e086baed8b2a78fb6db2d6cfe5e50d2ba6ffa5a069ccccf07e96c6ba31eb1f5 WatchSource:0}: Error finding container 2e086baed8b2a78fb6db2d6cfe5e50d2ba6ffa5a069ccccf07e96c6ba31eb1f5: Status 404 returned error can't find the container with id 2e086baed8b2a78fb6db2d6cfe5e50d2ba6ffa5a069ccccf07e96c6ba31eb1f5
Oct 31 07:51:25.634597 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:25.634554    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999" event={"ID":"b11e7548-483e-4939-ab37-f8e0410d7850","Type":"ContainerStarted","Data":"e91c31cc9ac45d0dd64344d6fe227cef73049f09ae20bff1f257cc4e14475362"}
Oct 31 07:51:25.635075 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:25.634618    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999" event={"ID":"b11e7548-483e-4939-ab37-f8e0410d7850","Type":"ContainerStarted","Data":"2e086baed8b2a78fb6db2d6cfe5e50d2ba6ffa5a069ccccf07e96c6ba31eb1f5"}
Oct 31 07:51:25.675090 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:25.674990    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999" podStartSLOduration=1.674976338 podStartE2EDuration="1.674976338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:51:25.673576924 +0000 UTC m=+1349.812737512" watchObservedRunningTime="2026-10-31 07:51:25.674976338 +0000 UTC m=+1349.814136922"
Oct 31 07:51:26.642792 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:26.642758    5829 generic.go:356] "Generic (PLEG): container finished" podID="b11e7548-483e-4939-ab37-f8e0410d7850" containerID="e91c31cc9ac45d0dd64344d6fe227cef73049f09ae20bff1f257cc4e14475362" exitCode=0
Oct 31 07:51:26.643207 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:26.642846    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999" event={"ID":"b11e7548-483e-4939-ab37-f8e0410d7850","Type":"ContainerDied","Data":"e91c31cc9ac45d0dd64344d6fe227cef73049f09ae20bff1f257cc4e14475362"}
Oct 31 07:51:27.785589 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.785557    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:27.887385 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.887343    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-xks-ca\") pod \"b11e7548-483e-4939-ab37-f8e0410d7850\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") "
Oct 31 07:51:27.887547 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.887416    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-openshift-ca\") pod \"b11e7548-483e-4939-ab37-f8e0410d7850\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") "
Oct 31 07:51:27.887547 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.887449    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11e7548-483e-4939-ab37-f8e0410d7850-kube-api-access-l5bjq\" (UniqueName: \"kubernetes.io/projected/b11e7548-483e-4939-ab37-f8e0410d7850-kube-api-access-l5bjq\") pod \"b11e7548-483e-4939-ab37-f8e0410d7850\" (UID: \"b11e7548-483e-4939-ab37-f8e0410d7850\") "
Oct 31 07:51:27.887829 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.887794    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-openshift-ca" pod "b11e7548-483e-4939-ab37-f8e0410d7850" (UID: "b11e7548-483e-4939-ab37-f8e0410d7850"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:51:27.889682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.889651    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-xks-ca" pod "b11e7548-483e-4939-ab37-f8e0410d7850" (UID: "b11e7548-483e-4939-ab37-f8e0410d7850"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:51:27.889769 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.889674    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11e7548-483e-4939-ab37-f8e0410d7850-kube-api-access-l5bjq" pod "b11e7548-483e-4939-ab37-f8e0410d7850" (UID: "b11e7548-483e-4939-ab37-f8e0410d7850"). InnerVolumeSpecName "kube-api-access-l5bjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:51:27.988884 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.988851    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:27.988884 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.988882    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11e7548-483e-4939-ab37-f8e0410d7850-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:27.989074 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:27.988892    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5bjq\" (UniqueName: \"kubernetes.io/projected/b11e7548-483e-4939-ab37-f8e0410d7850-kube-api-access-l5bjq\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:28.655149 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:28.655109    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"
Oct 31 07:51:28.655323 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:28.655117    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999" event={"ID":"b11e7548-483e-4939-ab37-f8e0410d7850","Type":"ContainerDied","Data":"2e086baed8b2a78fb6db2d6cfe5e50d2ba6ffa5a069ccccf07e96c6ba31eb1f5"}
Oct 31 07:51:28.655323 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:28.655236    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e086baed8b2a78fb6db2d6cfe5e50d2ba6ffa5a069ccccf07e96c6ba31eb1f5"
Oct 31 07:51:35.682556 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:35.682518    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:51:38.341522 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:38.341418    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-hkpt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:51:38.342434 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:38.342396    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:51:38.557430 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:38.557389    5829 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-645cf464d7-vt92f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 07:51:38.557620 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:38.557447    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 07:51:41.918188 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:41.918150    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:51:46.509218 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:51:46.509173    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:51:55.700780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.700708    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"]
Oct 31 07:51:55.713372 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.713337    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5g999"]
Oct 31 07:51:55.739553 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.739524    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"]
Oct 31 07:51:55.739974 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.739960    5829 cpu_manager.go:414] "containerMap: removing container" podUID="b11e7548-483e-4939-ab37-f8e0410d7850" containerName="revoke-keys"
Oct 31 07:51:55.740056 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.739974    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11e7548-483e-4939-ab37-f8e0410d7850" containerName="revoke-keys"
Oct 31 07:51:55.740119 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.740108    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11e7548-483e-4939-ab37-f8e0410d7850" containerName="revoke-keys"
Oct 31 07:51:55.743333 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.743314    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.760420 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.760392    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:51:55.760579 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.760522    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:51:55.760579 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.760560    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:51:55.761341 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.761316    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:51:55.763029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.763005    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"]
Oct 31 07:51:55.811097 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.811058    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78fd\" (UniqueName: \"kubernetes.io/projected/470f69de-eb8e-4d05-aaa6-5038180e5878-kube-api-access-d78fd\") pod \"maas-api-revoke-sub-3d4156f1-fppgp\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.811296 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.811122    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fppgp\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.811387 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.811355    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fppgp\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.912654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.912615    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78fd\" (UniqueName: \"kubernetes.io/projected/470f69de-eb8e-4d05-aaa6-5038180e5878-kube-api-access-d78fd\") pod \"maas-api-revoke-sub-3d4156f1-fppgp\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.912838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.912664    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fppgp\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.912838 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.912745    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fppgp\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.913493 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.913471    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fppgp\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.915125 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.915100    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fppgp\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.932193 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.932160    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78fd\" (UniqueName: \"kubernetes.io/projected/470f69de-eb8e-4d05-aaa6-5038180e5878-kube-api-access-d78fd\") pod \"maas-api-revoke-sub-3d4156f1-fppgp\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:55.972996 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:55.972913    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11e7548-483e-4939-ab37-f8e0410d7850" path="/var/lib/kubelet/pods/b11e7548-483e-4939-ab37-f8e0410d7850/volumes"
Oct 31 07:51:56.071060 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:56.071029    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:51:56.077077 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:56.077061    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:56.375186 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:56.375162    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"]
Oct 31 07:51:56.378026 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:51:56.377996    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470f69de_eb8e_4d05_aaa6_5038180e5878.slice/crio-ad184e425ec3a924f12202c6a8b4e11cd5a9fe86b8ef49cb2adaff2b51709380 WatchSource:0}: Error finding container ad184e425ec3a924f12202c6a8b4e11cd5a9fe86b8ef49cb2adaff2b51709380: Status 404 returned error can't find the container with id ad184e425ec3a924f12202c6a8b4e11cd5a9fe86b8ef49cb2adaff2b51709380
Oct 31 07:51:56.868512 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:56.868477    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp" event={"ID":"470f69de-eb8e-4d05-aaa6-5038180e5878","Type":"ContainerStarted","Data":"b1c4410cf1a0b6317736e567be8888ccecbe21a0d569d89d7d378c1e4c343c5f"}
Oct 31 07:51:56.868955 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:56.868529    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp" event={"ID":"470f69de-eb8e-4d05-aaa6-5038180e5878","Type":"ContainerStarted","Data":"ad184e425ec3a924f12202c6a8b4e11cd5a9fe86b8ef49cb2adaff2b51709380"}
Oct 31 07:51:56.908743 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:56.908669    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp" podStartSLOduration=1.908651699 podStartE2EDuration="1.908651699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:51:56.907305142 +0000 UTC m=+1381.046465720" watchObservedRunningTime="2026-10-31 07:51:56.908651699 +0000 UTC m=+1381.047812288"
Oct 31 07:51:57.875520 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:57.875483    5829 generic.go:356] "Generic (PLEG): container finished" podID="470f69de-eb8e-4d05-aaa6-5038180e5878" containerID="b1c4410cf1a0b6317736e567be8888ccecbe21a0d569d89d7d378c1e4c343c5f" exitCode=0
Oct 31 07:51:57.875944 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:57.875575    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp" event={"ID":"470f69de-eb8e-4d05-aaa6-5038180e5878","Type":"ContainerDied","Data":"b1c4410cf1a0b6317736e567be8888ccecbe21a0d569d89d7d378c1e4c343c5f"}
Oct 31 07:51:58.430220 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:58.430184    5829 scope.go:122] "RemoveContainer" containerID="e64c29233643d6caf2c8c56d216380fdfe31bc72528ad0dad1482f7c3b0403b0"
Oct 31 07:51:58.439412 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:58.439389    5829 scope.go:122] "RemoveContainer" containerID="009097d33939614b56f8ab35d3851c662499b8641bf0e9c0c3da342198320ded"
Oct 31 07:51:58.448004 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:58.447986    5829 scope.go:122] "RemoveContainer" containerID="65009fc9891eca7e1cc6522290b9bef9b8211fd6f59f38064f18465a2d76fc85"
Oct 31 07:51:58.456149 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:58.456130    5829 scope.go:122] "RemoveContainer" containerID="19c1dcf8e907286aa73372c942945c315f7de215d36f8c9dcd3d63f97e4a0c63"
Oct 31 07:51:58.464092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:58.464068    5829 scope.go:122] "RemoveContainer" containerID="d1d61c8970ef405e25ae6d4cff9c0b0d9617e08fa444f492a63c34b4d40bd7f7"
Oct 31 07:51:58.474684 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:58.474422    5829 scope.go:122] "RemoveContainer" containerID="3dec4a91bc4b01564357edd76664fb743dae73bc2822f94d458d0967d84fbf92"
Oct 31 07:51:58.483594 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:58.483576    5829 scope.go:122] "RemoveContainer" containerID="f90d365297092dc030152e04ba1dcf5a7e1f2270976ab82e617604512bf71482"
Oct 31 07:51:58.491409 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:58.491387    5829 scope.go:122] "RemoveContainer" containerID="7ebbcc2d31c7a0c6de306431ae0c4dba330957ee0b40b623df7de28d8612a3d5"
Oct 31 07:51:59.031030 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.031000    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:59.147803 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.147768    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-xks-ca\") pod \"470f69de-eb8e-4d05-aaa6-5038180e5878\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") "
Oct 31 07:51:59.147991 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.147817    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-openshift-ca\") pod \"470f69de-eb8e-4d05-aaa6-5038180e5878\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") "
Oct 31 07:51:59.147991 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.147873    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470f69de-eb8e-4d05-aaa6-5038180e5878-kube-api-access-d78fd\" (UniqueName: \"kubernetes.io/projected/470f69de-eb8e-4d05-aaa6-5038180e5878-kube-api-access-d78fd\") pod \"470f69de-eb8e-4d05-aaa6-5038180e5878\" (UID: \"470f69de-eb8e-4d05-aaa6-5038180e5878\") "
Oct 31 07:51:59.148256 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.148231    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-openshift-ca" pod "470f69de-eb8e-4d05-aaa6-5038180e5878" (UID: "470f69de-eb8e-4d05-aaa6-5038180e5878"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:51:59.149946 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.149917    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-xks-ca" pod "470f69de-eb8e-4d05-aaa6-5038180e5878" (UID: "470f69de-eb8e-4d05-aaa6-5038180e5878"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:51:59.150030 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.149971    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470f69de-eb8e-4d05-aaa6-5038180e5878-kube-api-access-d78fd" pod "470f69de-eb8e-4d05-aaa6-5038180e5878" (UID: "470f69de-eb8e-4d05-aaa6-5038180e5878"). InnerVolumeSpecName "kube-api-access-d78fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:51:59.249665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.249626    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:59.249665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.249660    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470f69de-eb8e-4d05-aaa6-5038180e5878-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:59.249665 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.249672    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78fd\" (UniqueName: \"kubernetes.io/projected/470f69de-eb8e-4d05-aaa6-5038180e5878-kube-api-access-d78fd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:51:59.889359 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.889319    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp" event={"ID":"470f69de-eb8e-4d05-aaa6-5038180e5878","Type":"ContainerDied","Data":"ad184e425ec3a924f12202c6a8b4e11cd5a9fe86b8ef49cb2adaff2b51709380"}
Oct 31 07:51:59.889359 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.889340    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"
Oct 31 07:51:59.889621 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:51:59.889374    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad184e425ec3a924f12202c6a8b4e11cd5a9fe86b8ef49cb2adaff2b51709380"
Oct 31 07:52:16.510884 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:52:16.510839    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:52:36.386701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:36.386662    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:52:43.125190 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.125149    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"]
Oct 31 07:52:43.125882 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.125862    5829 cpu_manager.go:414] "containerMap: removing container" podUID="470f69de-eb8e-4d05-aaa6-5038180e5878" containerName="revoke-keys"
Oct 31 07:52:43.125929 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.125885    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470f69de-eb8e-4d05-aaa6-5038180e5878" containerName="revoke-keys"
Oct 31 07:52:43.125997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.125984    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="470f69de-eb8e-4d05-aaa6-5038180e5878" containerName="revoke-keys"
Oct 31 07:52:43.129590 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.129572    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.145026 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.145002    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:52:43.145026 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.145022    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:52:43.158136 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.158097    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:52:43.158136 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.158100    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:52:43.160006 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.159979    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"]
Oct 31 07:52:43.226200 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.226161    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-244lv\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.226404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.226238    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-244lv\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.226404 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.226371    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g9ch\" (UniqueName: \"kubernetes.io/projected/8e5552e9-740e-4db9-a717-25c938fa6a37-kube-api-access-8g9ch\") pod \"maas-api-revoke-sub-e2fc9c5b-244lv\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.327113 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.327074    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-244lv\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.327336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.327144    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g9ch\" (UniqueName: \"kubernetes.io/projected/8e5552e9-740e-4db9-a717-25c938fa6a37-kube-api-access-8g9ch\") pod \"maas-api-revoke-sub-e2fc9c5b-244lv\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.327336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.327204    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-244lv\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.327842 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.327822    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-244lv\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.329637 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.329612    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-244lv\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.346946 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.346913    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g9ch\" (UniqueName: \"kubernetes.io/projected/8e5552e9-740e-4db9-a717-25c938fa6a37-kube-api-access-8g9ch\") pod \"maas-api-revoke-sub-e2fc9c5b-244lv\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.443340 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.443304    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:43.746250 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:43.746146    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"]
Oct 31 07:52:43.749345 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:52:43.749308    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5552e9_740e_4db9_a717_25c938fa6a37.slice/crio-b4329ea931f7721d5ff98cbfe4f0d4af21855cf0642043d71c7fc7f765632993 WatchSource:0}: Error finding container b4329ea931f7721d5ff98cbfe4f0d4af21855cf0642043d71c7fc7f765632993: Status 404 returned error can't find the container with id b4329ea931f7721d5ff98cbfe4f0d4af21855cf0642043d71c7fc7f765632993
Oct 31 07:52:44.131328 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:44.131271    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv" event={"ID":"8e5552e9-740e-4db9-a717-25c938fa6a37","Type":"ContainerStarted","Data":"9f8eddc956021c2bd30d746204b83ba50efd8b7029b7750c221cce8253aebf47"}
Oct 31 07:52:44.131799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:44.131343    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv" event={"ID":"8e5552e9-740e-4db9-a717-25c938fa6a37","Type":"ContainerStarted","Data":"b4329ea931f7721d5ff98cbfe4f0d4af21855cf0642043d71c7fc7f765632993"}
Oct 31 07:52:44.170295 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:44.170228    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv" podStartSLOduration=1.170207037 podStartE2EDuration="1.170207037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:52:44.169562507 +0000 UTC m=+1428.308723095" watchObservedRunningTime="2026-10-31 07:52:44.170207037 +0000 UTC m=+1428.309367622"
Oct 31 07:52:45.137926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:45.137887    5829 generic.go:356] "Generic (PLEG): container finished" podID="8e5552e9-740e-4db9-a717-25c938fa6a37" containerID="9f8eddc956021c2bd30d746204b83ba50efd8b7029b7750c221cce8253aebf47" exitCode=0
Oct 31 07:52:45.137926 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:45.137930    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv" event={"ID":"8e5552e9-740e-4db9-a717-25c938fa6a37","Type":"ContainerDied","Data":"9f8eddc956021c2bd30d746204b83ba50efd8b7029b7750c221cce8253aebf47"}
Oct 31 07:52:46.289474 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.289444    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:46.360438 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.360401    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-openshift-ca\") pod \"8e5552e9-740e-4db9-a717-25c938fa6a37\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") "
Oct 31 07:52:46.360624 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.360585    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5552e9-740e-4db9-a717-25c938fa6a37-kube-api-access-8g9ch\" (UniqueName: \"kubernetes.io/projected/8e5552e9-740e-4db9-a717-25c938fa6a37-kube-api-access-8g9ch\") pod \"8e5552e9-740e-4db9-a717-25c938fa6a37\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") "
Oct 31 07:52:46.360677 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.360649    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-xks-ca\") pod \"8e5552e9-740e-4db9-a717-25c938fa6a37\" (UID: \"8e5552e9-740e-4db9-a717-25c938fa6a37\") "
Oct 31 07:52:46.360791 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.360765    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-openshift-ca" pod "8e5552e9-740e-4db9-a717-25c938fa6a37" (UID: "8e5552e9-740e-4db9-a717-25c938fa6a37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:52:46.360947 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.360932    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:52:46.362780 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.362747    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-xks-ca" pod "8e5552e9-740e-4db9-a717-25c938fa6a37" (UID: "8e5552e9-740e-4db9-a717-25c938fa6a37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:52:46.362879 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.362806    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5552e9-740e-4db9-a717-25c938fa6a37-kube-api-access-8g9ch" pod "8e5552e9-740e-4db9-a717-25c938fa6a37" (UID: "8e5552e9-740e-4db9-a717-25c938fa6a37"). InnerVolumeSpecName "kube-api-access-8g9ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:52:46.462182 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.462096    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g9ch\" (UniqueName: \"kubernetes.io/projected/8e5552e9-740e-4db9-a717-25c938fa6a37-kube-api-access-8g9ch\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:52:46.462182 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:46.462129    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e5552e9-740e-4db9-a717-25c938fa6a37-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:52:46.510562 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:52:46.510519    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:52:47.149952 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:47.149922    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"
Oct 31 07:52:47.150143 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:47.149953    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv" event={"ID":"8e5552e9-740e-4db9-a717-25c938fa6a37","Type":"ContainerDied","Data":"b4329ea931f7721d5ff98cbfe4f0d4af21855cf0642043d71c7fc7f765632993"}
Oct 31 07:52:47.150143 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:47.150012    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4329ea931f7721d5ff98cbfe4f0d4af21855cf0642043d71c7fc7f765632993"
Oct 31 07:52:53.441616 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.441583    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"]
Oct 31 07:52:53.442085 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.442025    5829 cpu_manager.go:414] "containerMap: removing container" podUID="8e5552e9-740e-4db9-a717-25c938fa6a37" containerName="revoke-keys"
Oct 31 07:52:53.442085 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.442035    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5552e9-740e-4db9-a717-25c938fa6a37" containerName="revoke-keys"
Oct 31 07:52:53.442157 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.442129    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5552e9-740e-4db9-a717-25c938fa6a37" containerName="revoke-keys"
Oct 31 07:52:53.445654 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.445631    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.478753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.478722    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:52:53.478957 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.478759    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:52:53.479044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.478771    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:52:53.479044 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.478820    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:52:53.480910 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.480889    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"]
Oct 31 07:52:53.535193 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.535154    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-r5xh6\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.535400 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.535325    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f945\" (UniqueName: \"kubernetes.io/projected/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-kube-api-access-4f945\") pod \"maas-api-revoke-sub-5a29efb3-r5xh6\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.535447 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.535431    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-r5xh6\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.636578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.636541    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f945\" (UniqueName: \"kubernetes.io/projected/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-kube-api-access-4f945\") pod \"maas-api-revoke-sub-5a29efb3-r5xh6\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.636753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.636609    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-r5xh6\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.636753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.636653    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-r5xh6\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.637254 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.637225    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-r5xh6\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.639050 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.639024    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-r5xh6\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.656350 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.656323    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f945\" (UniqueName: \"kubernetes.io/projected/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-kube-api-access-4f945\") pod \"maas-api-revoke-sub-5a29efb3-r5xh6\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:53.761035 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:53.760951    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:54.058871 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:54.058841    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"]
Oct 31 07:52:54.061833 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:52:54.061798    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac7ea7f_fa0e_4922_8e87_2f239cc5be5a.slice/crio-60199fd1fe423dca0ff531849d3e2066ae225e53fb78d26ee7fcfd6201343556 WatchSource:0}: Error finding container 60199fd1fe423dca0ff531849d3e2066ae225e53fb78d26ee7fcfd6201343556: Status 404 returned error can't find the container with id 60199fd1fe423dca0ff531849d3e2066ae225e53fb78d26ee7fcfd6201343556
Oct 31 07:52:54.187025 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:54.186985    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6" event={"ID":"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a","Type":"ContainerStarted","Data":"60199fd1fe423dca0ff531849d3e2066ae225e53fb78d26ee7fcfd6201343556"}
Oct 31 07:52:54.445225 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:54.445189    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:52:55.195987 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:55.195947    5829 generic.go:356] "Generic (PLEG): container finished" podID="7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a" containerID="2d275a51ba73718d25fa0451762f411206115b8ffad50e7c8031f53809e66752" exitCode=0
Oct 31 07:52:55.196203 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:55.196011    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6" event={"ID":"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a","Type":"ContainerDied","Data":"2d275a51ba73718d25fa0451762f411206115b8ffad50e7c8031f53809e66752"}
Oct 31 07:52:56.630312 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.613480    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:56.799865 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.799834    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-openshift-ca\") pod \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") "
Oct 31 07:52:56.800042 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.799981    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-xks-ca\") pod \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") "
Oct 31 07:52:56.800146 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.800129    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-kube-api-access-4f945\" (UniqueName: \"kubernetes.io/projected/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-kube-api-access-4f945\") pod \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\" (UID: \"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a\") "
Oct 31 07:52:56.800240 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.800219    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-openshift-ca" pod "7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a" (UID: "7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:52:56.800487 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.800470    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:52:56.802076 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.802054    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-kube-api-access-4f945" pod "7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a" (UID: "7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a"). InnerVolumeSpecName "kube-api-access-4f945". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:52:56.802147 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.802086    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-xks-ca" pod "7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a" (UID: "7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:52:56.901296 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.901244    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:52:56.901471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:56.901305    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f945\" (UniqueName: \"kubernetes.io/projected/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a-kube-api-access-4f945\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:52:57.212364 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:57.211391    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6" event={"ID":"7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a","Type":"ContainerDied","Data":"60199fd1fe423dca0ff531849d3e2066ae225e53fb78d26ee7fcfd6201343556"}
Oct 31 07:52:57.212364 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:57.211473    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60199fd1fe423dca0ff531849d3e2066ae225e53fb78d26ee7fcfd6201343556"
Oct 31 07:52:57.212364 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:57.211570    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"
Oct 31 07:52:58.542413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:52:58.542379    5829 scope.go:122] "RemoveContainer" containerID="21e9f0dfc201ce739738f849faca17d6689a7b8b9d5bc89fb5ce65d8126dc5a4"
Oct 31 07:53:04.758639 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:04.758596    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:53:16.510593 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:53:16.510511    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:53:19.307471 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.307430    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"]
Oct 31 07:53:19.320125 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.320092    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fppgp"]
Oct 31 07:53:19.352089 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.352051    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"]
Oct 31 07:53:19.352624 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.352606    5829 cpu_manager.go:414] "containerMap: removing container" podUID="7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a" containerName="revoke-keys"
Oct 31 07:53:19.352676 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.352627    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a" containerName="revoke-keys"
Oct 31 07:53:19.352727 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.352717    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a" containerName="revoke-keys"
Oct 31 07:53:19.368249 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.368198    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.375997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.375963    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"]
Oct 31 07:53:19.388317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.384050    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:53:19.388317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.384150    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:53:19.388317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.384228    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:53:19.388317 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.384255    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:53:19.431682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.431640    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5mg6p\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.431878 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.431746    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9bsh\" (UniqueName: \"kubernetes.io/projected/997d8fe6-d818-47e5-8326-150a9f761014-kube-api-access-t9bsh\") pod \"maas-api-revoke-sub-3d4156f1-5mg6p\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.431878 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.431839    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5mg6p\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.532724 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.532676    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5mg6p\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.532915 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.532787    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5mg6p\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.532915 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.532830    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9bsh\" (UniqueName: \"kubernetes.io/projected/997d8fe6-d818-47e5-8326-150a9f761014-kube-api-access-t9bsh\") pod \"maas-api-revoke-sub-3d4156f1-5mg6p\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.533535 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.533512    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5mg6p\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.535138 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.535115    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5mg6p\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.552586 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.552553    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9bsh\" (UniqueName: \"kubernetes.io/projected/997d8fe6-d818-47e5-8326-150a9f761014-kube-api-access-t9bsh\") pod \"maas-api-revoke-sub-3d4156f1-5mg6p\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.685452 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.685417    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:19.974755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.974661    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470f69de-eb8e-4d05-aaa6-5038180e5878" path="/var/lib/kubelet/pods/470f69de-eb8e-4d05-aaa6-5038180e5878/volumes"
Oct 31 07:53:19.982513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:19.982485    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"]
Oct 31 07:53:19.985199 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:53:19.985169    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997d8fe6_d818_47e5_8326_150a9f761014.slice/crio-ff2ee64b8c4eedee2f5d8e567112768369cca86e58c45639ba22acc8f8ef2c24 WatchSource:0}: Error finding container ff2ee64b8c4eedee2f5d8e567112768369cca86e58c45639ba22acc8f8ef2c24: Status 404 returned error can't find the container with id ff2ee64b8c4eedee2f5d8e567112768369cca86e58c45639ba22acc8f8ef2c24
Oct 31 07:53:20.337724 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:20.337692    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p" event={"ID":"997d8fe6-d818-47e5-8326-150a9f761014","Type":"ContainerStarted","Data":"b4168c65863279a5ab4f76ddc1794c2abbc46e10d7abe1d09cb27238d4bfa1f7"}
Oct 31 07:53:20.338145 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:20.337742    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p" event={"ID":"997d8fe6-d818-47e5-8326-150a9f761014","Type":"ContainerStarted","Data":"ff2ee64b8c4eedee2f5d8e567112768369cca86e58c45639ba22acc8f8ef2c24"}
Oct 31 07:53:20.376922 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:20.376862    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p" podStartSLOduration=1.376847598 podStartE2EDuration="1.376847598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:53:20.375705905 +0000 UTC m=+1464.514866493" watchObservedRunningTime="2026-10-31 07:53:20.376847598 +0000 UTC m=+1464.516008186"
Oct 31 07:53:21.345065 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:21.344974    5829 generic.go:356] "Generic (PLEG): container finished" podID="997d8fe6-d818-47e5-8326-150a9f761014" containerID="b4168c65863279a5ab4f76ddc1794c2abbc46e10d7abe1d09cb27238d4bfa1f7" exitCode=0
Oct 31 07:53:21.345489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:21.345061    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p" event={"ID":"997d8fe6-d818-47e5-8326-150a9f761014","Type":"ContainerDied","Data":"b4168c65863279a5ab4f76ddc1794c2abbc46e10d7abe1d09cb27238d4bfa1f7"}
Oct 31 07:53:22.495797 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.495768    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:22.563344 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.563308    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-xks-ca\") pod \"997d8fe6-d818-47e5-8326-150a9f761014\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") "
Oct 31 07:53:22.563570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.563455    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997d8fe6-d818-47e5-8326-150a9f761014-kube-api-access-t9bsh\" (UniqueName: \"kubernetes.io/projected/997d8fe6-d818-47e5-8326-150a9f761014-kube-api-access-t9bsh\") pod \"997d8fe6-d818-47e5-8326-150a9f761014\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") "
Oct 31 07:53:22.563570 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.563517    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-openshift-ca\") pod \"997d8fe6-d818-47e5-8326-150a9f761014\" (UID: \"997d8fe6-d818-47e5-8326-150a9f761014\") "
Oct 31 07:53:22.563967 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.563931    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-openshift-ca" pod "997d8fe6-d818-47e5-8326-150a9f761014" (UID: "997d8fe6-d818-47e5-8326-150a9f761014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:53:22.565575 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.565545    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997d8fe6-d818-47e5-8326-150a9f761014-kube-api-access-t9bsh" pod "997d8fe6-d818-47e5-8326-150a9f761014" (UID: "997d8fe6-d818-47e5-8326-150a9f761014"). InnerVolumeSpecName "kube-api-access-t9bsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:53:22.565701 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.565632    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-xks-ca" pod "997d8fe6-d818-47e5-8326-150a9f761014" (UID: "997d8fe6-d818-47e5-8326-150a9f761014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:53:22.665336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.665307    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:53:22.665336 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.665335    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9bsh\" (UniqueName: \"kubernetes.io/projected/997d8fe6-d818-47e5-8326-150a9f761014-kube-api-access-t9bsh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:53:22.665546 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:22.665346    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997d8fe6-d818-47e5-8326-150a9f761014-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:53:23.358970 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:23.358944    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"
Oct 31 07:53:23.359144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:23.358937    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p" event={"ID":"997d8fe6-d818-47e5-8326-150a9f761014","Type":"ContainerDied","Data":"ff2ee64b8c4eedee2f5d8e567112768369cca86e58c45639ba22acc8f8ef2c24"}
Oct 31 07:53:23.359144 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:23.359067    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff2ee64b8c4eedee2f5d8e567112768369cca86e58c45639ba22acc8f8ef2c24"
Oct 31 07:53:46.510079 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:53:46.510035    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:53:50.370741 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.370705    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"]
Oct 31 07:53:50.371206 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.371170    5829 cpu_manager.go:414] "containerMap: removing container" podUID="997d8fe6-d818-47e5-8326-150a9f761014" containerName="revoke-keys"
Oct 31 07:53:50.371206 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.371186    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997d8fe6-d818-47e5-8326-150a9f761014" containerName="revoke-keys"
Oct 31 07:53:50.371373 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.371348    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="997d8fe6-d818-47e5-8326-150a9f761014" containerName="revoke-keys"
Oct 31 07:53:50.374129 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.374113    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:53:50.404778 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.404748    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-009b0f\""
Oct 31 07:53:50.406783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.406758    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"]
Oct 31 07:53:50.419500 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.419469    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"]
Oct 31 07:53:50.423426 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.423409    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:53:50.435984 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.435963    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-4a34b628-dockercfg-d87gh\""
Oct 31 07:53:50.447048 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.447020    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"]
Oct 31 07:53:50.454971 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.454944    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwjv\" (UniqueName: \"kubernetes.io/projected/25954d97-3fc0-4399-a7de-9e843a41c549-kube-api-access-vnwjv\") pod \"payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk\" (UID: \"25954d97-3fc0-4399-a7de-9e843a41c549\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:53:50.455118 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.454987    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25954d97-3fc0-4399-a7de-9e843a41c549-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk\" (UID: \"25954d97-3fc0-4399-a7de-9e843a41c549\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:53:50.555958 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.555923    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwjv\" (UniqueName: \"kubernetes.io/projected/25954d97-3fc0-4399-a7de-9e843a41c549-kube-api-access-vnwjv\") pod \"payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk\" (UID: \"25954d97-3fc0-4399-a7de-9e843a41c549\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:53:50.556135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.555967    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25954d97-3fc0-4399-a7de-9e843a41c549-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk\" (UID: \"25954d97-3fc0-4399-a7de-9e843a41c549\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:53:50.556135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.556008    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs\" (UID: \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:53:50.556135 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.556038    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8hr5\" (UniqueName: \"kubernetes.io/projected/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-kube-api-access-m8hr5\") pod \"payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs\" (UID: \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:53:50.556745 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.556726    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25954d97-3fc0-4399-a7de-9e843a41c549-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk\" (UID: \"25954d97-3fc0-4399-a7de-9e843a41c549\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:53:50.577413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.577384    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwjv\" (UniqueName: \"kubernetes.io/projected/25954d97-3fc0-4399-a7de-9e843a41c549-kube-api-access-vnwjv\") pod \"payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk\" (UID: \"25954d97-3fc0-4399-a7de-9e843a41c549\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:53:50.656988 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.656950    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs\" (UID: \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:53:50.657193 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.657009    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8hr5\" (UniqueName: \"kubernetes.io/projected/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-kube-api-access-m8hr5\") pod \"payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs\" (UID: \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:53:50.657764 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.657739    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs\" (UID: \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:53:50.676943 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.676911    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8hr5\" (UniqueName: \"kubernetes.io/projected/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-kube-api-access-m8hr5\") pod \"payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs\" (UID: \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:53:50.688813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.688786    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:53:50.737856 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.737310    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:53:50.986949 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:50.986925    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"]
Oct 31 07:53:50.989156 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:53:50.989125    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25954d97_3fc0_4399_a7de_9e843a41c549.slice/crio-d301e832fe1369b1249b1f7976507fcf19217955153fcde9a687998a7a528505 WatchSource:0}: Error finding container d301e832fe1369b1249b1f7976507fcf19217955153fcde9a687998a7a528505: Status 404 returned error can't find the container with id d301e832fe1369b1249b1f7976507fcf19217955153fcde9a687998a7a528505
Oct 31 07:53:51.047398 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:51.047369    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"]
Oct 31 07:53:51.049604 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:53:51.049574    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f47102_5e2c_4f62_8cfb_2adf62f1a474.slice/crio-2179abcfafef5212f12282ba6154bcc0f2f600f47db8bec9ceae0f67394cca71 WatchSource:0}: Error finding container 2179abcfafef5212f12282ba6154bcc0f2f600f47db8bec9ceae0f67394cca71: Status 404 returned error can't find the container with id 2179abcfafef5212f12282ba6154bcc0f2f600f47db8bec9ceae0f67394cca71
Oct 31 07:53:51.516475 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:51.516438    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs" event={"ID":"b1f47102-5e2c-4f62-8cfb-2adf62f1a474","Type":"ContainerStarted","Data":"728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685"}
Oct 31 07:53:51.516931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:51.516507    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:53:51.516931 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:51.516533    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs" event={"ID":"b1f47102-5e2c-4f62-8cfb-2adf62f1a474","Type":"ContainerStarted","Data":"2179abcfafef5212f12282ba6154bcc0f2f600f47db8bec9ceae0f67394cca71"}
Oct 31 07:53:51.518195 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:51.518161    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk" event={"ID":"25954d97-3fc0-4399-a7de-9e843a41c549","Type":"ContainerStarted","Data":"dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0"}
Oct 31 07:53:51.518343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:51.518210    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk" event={"ID":"25954d97-3fc0-4399-a7de-9e843a41c549","Type":"ContainerStarted","Data":"d301e832fe1369b1249b1f7976507fcf19217955153fcde9a687998a7a528505"}
Oct 31 07:53:51.518343 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:51.518230    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:53:51.558061 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:51.558003    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs" podStartSLOduration=1.5579879330000002 podStartE2EDuration="1.557987933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:53:51.555397045 +0000 UTC m=+1495.694557633" watchObservedRunningTime="2026-10-31 07:53:51.557987933 +0000 UTC m=+1495.697148560"
Oct 31 07:53:51.596521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:51.596465    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk" podStartSLOduration=1.596449766 podStartE2EDuration="1.596449766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:53:51.59458193 +0000 UTC m=+1495.733742517" watchObservedRunningTime="2026-10-31 07:53:51.596449766 +0000 UTC m=+1495.735610354"
Oct 31 07:53:53.762319 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:53.762260    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:53:58.111339 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:58.111308    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:53:58.111836 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:53:58.111360    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:54:02.526322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:02.526272    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:54:02.526852 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:02.526499    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:54:16.514407 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:54:16.514265    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:54:18.398384 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:18.398348    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:54:33.881834 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:33.881792    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:54:46.510510 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:54:46.510416    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:54:47.757366 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.757337    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"]
Oct 31 07:54:47.761413 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.761385    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:47.762890 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.762862    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ab697c-jlkkn\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:47.763040 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.763013    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk8rd\" (UniqueName: \"kubernetes.io/projected/9d576fc9-635e-40bb-be85-6d0dc1955730-kube-api-access-kk8rd\") pod \"maas-api-revoke-sub-e8ab697c-jlkkn\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:47.763176 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.763158    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ab697c-jlkkn\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:47.776990 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.776942    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:54:47.777146 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.777039    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:54:47.789984 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.789954    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:54:47.789984 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.789988    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:54:47.791997 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.791972    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"]
Oct 31 07:54:47.864208 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.864173    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk8rd\" (UniqueName: \"kubernetes.io/projected/9d576fc9-635e-40bb-be85-6d0dc1955730-kube-api-access-kk8rd\") pod \"maas-api-revoke-sub-e8ab697c-jlkkn\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:47.864412 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.864323    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ab697c-jlkkn\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:47.864412 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.864363    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ab697c-jlkkn\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:47.865050 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.865004    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ab697c-jlkkn\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:47.867053 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.867031    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ab697c-jlkkn\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:47.884174 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:47.884145    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk8rd\" (UniqueName: \"kubernetes.io/projected/9d576fc9-635e-40bb-be85-6d0dc1955730-kube-api-access-kk8rd\") pod \"maas-api-revoke-sub-e8ab697c-jlkkn\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:48.076545 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:48.076444    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:48.375666 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:48.375562    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"]
Oct 31 07:54:48.379132 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:54:48.379088    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d576fc9_635e_40bb_be85_6d0dc1955730.slice/crio-df6d64841e34e841b8a324e4944a065401494b31840a3e7f5ed61e71ce89f94f WatchSource:0}: Error finding container df6d64841e34e841b8a324e4944a065401494b31840a3e7f5ed61e71ce89f94f: Status 404 returned error can't find the container with id df6d64841e34e841b8a324e4944a065401494b31840a3e7f5ed61e71ce89f94f
Oct 31 07:54:48.851142 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:48.851097    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn" event={"ID":"9d576fc9-635e-40bb-be85-6d0dc1955730","Type":"ContainerStarted","Data":"674e4b9c352380df98f8ba61f4b473e3e6b4f61081acbe95f4824781295e310b"}
Oct 31 07:54:48.851597 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:48.851165    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn" event={"ID":"9d576fc9-635e-40bb-be85-6d0dc1955730","Type":"ContainerStarted","Data":"df6d64841e34e841b8a324e4944a065401494b31840a3e7f5ed61e71ce89f94f"}
Oct 31 07:54:48.891403 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:48.891346    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn" podStartSLOduration=1.891330409 podStartE2EDuration="1.891330409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:54:48.889965019 +0000 UTC m=+1553.029125608" watchObservedRunningTime="2026-10-31 07:54:48.891330409 +0000 UTC m=+1553.030490998"
Oct 31 07:54:49.862450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:49.862412    5829 generic.go:356] "Generic (PLEG): container finished" podID="9d576fc9-635e-40bb-be85-6d0dc1955730" containerID="674e4b9c352380df98f8ba61f4b473e3e6b4f61081acbe95f4824781295e310b" exitCode=0
Oct 31 07:54:49.862866 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:49.862500    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn" event={"ID":"9d576fc9-635e-40bb-be85-6d0dc1955730","Type":"ContainerDied","Data":"674e4b9c352380df98f8ba61f4b473e3e6b4f61081acbe95f4824781295e310b"}
Oct 31 07:54:51.010933 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.010903    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:51.090675 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.090640    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d576fc9-635e-40bb-be85-6d0dc1955730-kube-api-access-kk8rd\" (UniqueName: \"kubernetes.io/projected/9d576fc9-635e-40bb-be85-6d0dc1955730-kube-api-access-kk8rd\") pod \"9d576fc9-635e-40bb-be85-6d0dc1955730\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") "
Oct 31 07:54:51.090880 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.090717    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-xks-ca\") pod \"9d576fc9-635e-40bb-be85-6d0dc1955730\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") "
Oct 31 07:54:51.090880 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.090799    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-openshift-ca\") pod \"9d576fc9-635e-40bb-be85-6d0dc1955730\" (UID: \"9d576fc9-635e-40bb-be85-6d0dc1955730\") "
Oct 31 07:54:51.091206 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.091180    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-openshift-ca" pod "9d576fc9-635e-40bb-be85-6d0dc1955730" (UID: "9d576fc9-635e-40bb-be85-6d0dc1955730"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:54:51.092963 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.092936    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-xks-ca" pod "9d576fc9-635e-40bb-be85-6d0dc1955730" (UID: "9d576fc9-635e-40bb-be85-6d0dc1955730"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:54:51.093078 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.093018    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d576fc9-635e-40bb-be85-6d0dc1955730-kube-api-access-kk8rd" pod "9d576fc9-635e-40bb-be85-6d0dc1955730" (UID: "9d576fc9-635e-40bb-be85-6d0dc1955730"). InnerVolumeSpecName "kube-api-access-kk8rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:54:51.191755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.191719    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk8rd\" (UniqueName: \"kubernetes.io/projected/9d576fc9-635e-40bb-be85-6d0dc1955730-kube-api-access-kk8rd\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:54:51.191755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.191751    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:54:51.191755 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.191761    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d576fc9-635e-40bb-be85-6d0dc1955730-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:54:51.874720 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.874688    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"
Oct 31 07:54:51.874720 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.874700    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn" event={"ID":"9d576fc9-635e-40bb-be85-6d0dc1955730","Type":"ContainerDied","Data":"df6d64841e34e841b8a324e4944a065401494b31840a3e7f5ed61e71ce89f94f"}
Oct 31 07:54:51.874938 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:51.874744    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6d64841e34e841b8a324e4944a065401494b31840a3e7f5ed61e71ce89f94f"
Oct 31 07:54:59.994666 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:54:59.994630    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:55:08.921714 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.921676    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"]
Oct 31 07:55:08.922326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.922296    5829 cpu_manager.go:414] "containerMap: removing container" podUID="9d576fc9-635e-40bb-be85-6d0dc1955730" containerName="revoke-keys"
Oct 31 07:55:08.922326 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.922321    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d576fc9-635e-40bb-be85-6d0dc1955730" containerName="revoke-keys"
Oct 31 07:55:08.922498 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.922484    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d576fc9-635e-40bb-be85-6d0dc1955730" containerName="revoke-keys"
Oct 31 07:55:08.926092 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.926077    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:08.942259 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.942229    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:55:08.955052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.955024    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:55:08.955052 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.955027    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:55:08.955314 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.955078    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:55:08.957380 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.957356    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"]
Oct 31 07:55:08.987085 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.987042    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:08.987274 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.987097    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69d9l\" (UniqueName: \"kubernetes.io/projected/33875a37-83c7-4b4a-b649-66deabad5179-kube-api-access-69d9l\") pod \"maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:08.987274 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:08.987232    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:09.087946 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.087908    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:09.088142 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.088015    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:09.088142 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.088047    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69d9l\" (UniqueName: \"kubernetes.io/projected/33875a37-83c7-4b4a-b649-66deabad5179-kube-api-access-69d9l\") pod \"maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:09.089101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.089054    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:09.098134 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.098098    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:09.108162 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.108127    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69d9l\" (UniqueName: \"kubernetes.io/projected/33875a37-83c7-4b4a-b649-66deabad5179-kube-api-access-69d9l\") pod \"maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:09.239168 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.239076    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:09.537754 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.537723    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"]
Oct 31 07:55:09.540049 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:55:09.540020    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33875a37_83c7_4b4a_b649_66deabad5179.slice/crio-76ca5bdc73a819c3576d567b4924204bddf4203771ab97842d4af76b0060a10d WatchSource:0}: Error finding container 76ca5bdc73a819c3576d567b4924204bddf4203771ab97842d4af76b0060a10d: Status 404 returned error can't find the container with id 76ca5bdc73a819c3576d567b4924204bddf4203771ab97842d4af76b0060a10d
Oct 31 07:55:09.981255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.981217    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df" event={"ID":"33875a37-83c7-4b4a-b649-66deabad5179","Type":"ContainerStarted","Data":"f13a7b8b5404f351b19caa1165b839f3f8de01a3cc2a35db16e41d1e4b11c856"}
Oct 31 07:55:09.981681 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:09.981271    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df" event={"ID":"33875a37-83c7-4b4a-b649-66deabad5179","Type":"ContainerStarted","Data":"76ca5bdc73a819c3576d567b4924204bddf4203771ab97842d4af76b0060a10d"}
Oct 31 07:55:10.020588 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:10.020530    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df" podStartSLOduration=2.020515576 podStartE2EDuration="2.020515576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:55:10.019129188 +0000 UTC m=+1574.158289776" watchObservedRunningTime="2026-10-31 07:55:10.020515576 +0000 UTC m=+1574.159676163"
Oct 31 07:55:10.988323 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:10.988265    5829 generic.go:356] "Generic (PLEG): container finished" podID="33875a37-83c7-4b4a-b649-66deabad5179" containerID="f13a7b8b5404f351b19caa1165b839f3f8de01a3cc2a35db16e41d1e4b11c856" exitCode=0
Oct 31 07:55:10.988739 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:10.988348    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df" event={"ID":"33875a37-83c7-4b4a-b649-66deabad5179","Type":"ContainerDied","Data":"f13a7b8b5404f351b19caa1165b839f3f8de01a3cc2a35db16e41d1e4b11c856"}
Oct 31 07:55:12.136425 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.136398    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:12.223006 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.222972    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-xks-ca\") pod \"33875a37-83c7-4b4a-b649-66deabad5179\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") "
Oct 31 07:55:12.223167 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.223109    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33875a37-83c7-4b4a-b649-66deabad5179-kube-api-access-69d9l\" (UniqueName: \"kubernetes.io/projected/33875a37-83c7-4b4a-b649-66deabad5179-kube-api-access-69d9l\") pod \"33875a37-83c7-4b4a-b649-66deabad5179\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") "
Oct 31 07:55:12.223210 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.223199    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-openshift-ca\") pod \"33875a37-83c7-4b4a-b649-66deabad5179\" (UID: \"33875a37-83c7-4b4a-b649-66deabad5179\") "
Oct 31 07:55:12.223576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.223545    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-openshift-ca" pod "33875a37-83c7-4b4a-b649-66deabad5179" (UID: "33875a37-83c7-4b4a-b649-66deabad5179"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:55:12.225784 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.225751    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-xks-ca" pod "33875a37-83c7-4b4a-b649-66deabad5179" (UID: "33875a37-83c7-4b4a-b649-66deabad5179"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:55:12.225877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.225814    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33875a37-83c7-4b4a-b649-66deabad5179-kube-api-access-69d9l" pod "33875a37-83c7-4b4a-b649-66deabad5179" (UID: "33875a37-83c7-4b4a-b649-66deabad5179"). InnerVolumeSpecName "kube-api-access-69d9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:55:12.324428 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.324330    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69d9l\" (UniqueName: \"kubernetes.io/projected/33875a37-83c7-4b4a-b649-66deabad5179-kube-api-access-69d9l\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:12.324428 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.324365    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:12.324428 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:12.324374    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33875a37-83c7-4b4a-b649-66deabad5179-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:13.000134 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:13.000100    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"
Oct 31 07:55:13.000353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:13.000143    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df" event={"ID":"33875a37-83c7-4b4a-b649-66deabad5179","Type":"ContainerDied","Data":"76ca5bdc73a819c3576d567b4924204bddf4203771ab97842d4af76b0060a10d"}
Oct 31 07:55:13.000353 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:13.000194    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76ca5bdc73a819c3576d567b4924204bddf4203771ab97842d4af76b0060a10d"
Oct 31 07:55:16.511462 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:55:16.511422    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:55:19.327016 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:19.326971    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"]
Oct 31 07:55:19.357482 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:19.357436    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"]
Oct 31 07:55:21.117374 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:21.117337    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"]
Oct 31 07:55:21.130436 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:21.130403    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-009b0f-gc7df"]
Oct 31 07:55:21.973400 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:21.973365    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33875a37-83c7-4b4a-b649-66deabad5179" path="/var/lib/kubelet/pods/33875a37-83c7-4b4a-b649-66deabad5179/volumes"
Oct 31 07:55:22.662883 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:22.662829    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:55:24.327857 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:24.327798    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk" podUID="25954d97-3fc0-4399-a7de-9e843a41c549" containerName="payload-processing" containerID="cri-o://dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0" gracePeriod=25
Oct 31 07:55:24.357996 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:24.357950    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs" podUID="b1f47102-5e2c-4f62-8cfb-2adf62f1a474" containerName="payload-pre-processing" containerID="cri-o://728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685" gracePeriod=25
Oct 31 07:55:32.525322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:32.525245    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs" podUID="b1f47102-5e2c-4f62-8cfb-2adf62f1a474" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:55:32.525322 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:32.525307    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk" podUID="25954d97-3fc0-4399-a7de-9e843a41c549" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:55:38.906173 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.906133    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"]
Oct 31 07:55:38.919108 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.919071    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5mg6p"]
Oct 31 07:55:38.951502 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.951461    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"]
Oct 31 07:55:38.951998 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.951983    5829 cpu_manager.go:414] "containerMap: removing container" podUID="33875a37-83c7-4b4a-b649-66deabad5179" containerName="revoke-keys"
Oct 31 07:55:38.952047 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.952000    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33875a37-83c7-4b4a-b649-66deabad5179" containerName="revoke-keys"
Oct 31 07:55:38.952098 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.952088    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="33875a37-83c7-4b4a-b649-66deabad5179" containerName="revoke-keys"
Oct 31 07:55:38.955733 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.955715    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:38.972094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.972031    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:55:38.972094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.972059    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:55:38.972094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.972084    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:55:38.972441 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.972403    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:55:38.972662 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:38.972643    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"]
Oct 31 07:55:39.004013 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.003966    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sftn4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.004013 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.004015    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jvwn\" (UniqueName: \"kubernetes.io/projected/d3c59a7a-9762-4473-953c-bb8ae815eac4-kube-api-access-5jvwn\") pod \"maas-api-revoke-sub-3d4156f1-sftn4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.004298 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.004042    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sftn4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.105339 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.105303    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jvwn\" (UniqueName: \"kubernetes.io/projected/d3c59a7a-9762-4473-953c-bb8ae815eac4-kube-api-access-5jvwn\") pod \"maas-api-revoke-sub-3d4156f1-sftn4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.105519 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.105348    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sftn4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.105519 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.105499    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sftn4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.106189 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.106164    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-sftn4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.107856 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.107834    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-sftn4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.125099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.125070    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jvwn\" (UniqueName: \"kubernetes.io/projected/d3c59a7a-9762-4473-953c-bb8ae815eac4-kube-api-access-5jvwn\") pod \"maas-api-revoke-sub-3d4156f1-sftn4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.268682 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.268597    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:39.567886 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.567861    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"]
Oct 31 07:55:39.570151 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:55:39.570116    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c59a7a_9762_4473_953c_bb8ae815eac4.slice/crio-ec7bd5300ad78c0a8bb8eae458822595fc43af74a40d2669b91d491c072faa64 WatchSource:0}: Error finding container ec7bd5300ad78c0a8bb8eae458822595fc43af74a40d2669b91d491c072faa64: Status 404 returned error can't find the container with id ec7bd5300ad78c0a8bb8eae458822595fc43af74a40d2669b91d491c072faa64
Oct 31 07:55:39.974410 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:39.974371    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997d8fe6-d818-47e5-8326-150a9f761014" path="/var/lib/kubelet/pods/997d8fe6-d818-47e5-8326-150a9f761014/volumes"
Oct 31 07:55:40.157941 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:40.157902    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4" event={"ID":"d3c59a7a-9762-4473-953c-bb8ae815eac4","Type":"ContainerStarted","Data":"1a526e5b697ddba34dd02cd4ebd12aa8e6394456bb1c36d5ced00bfebe0d3f31"}
Oct 31 07:55:40.158122 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:40.157960    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4" event={"ID":"d3c59a7a-9762-4473-953c-bb8ae815eac4","Type":"ContainerStarted","Data":"ec7bd5300ad78c0a8bb8eae458822595fc43af74a40d2669b91d491c072faa64"}
Oct 31 07:55:40.198538 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:40.198429    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4" podStartSLOduration=2.198414192 podStartE2EDuration="2.198414192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:55:40.196934286 +0000 UTC m=+1604.336094873" watchObservedRunningTime="2026-10-31 07:55:40.198414192 +0000 UTC m=+1604.337574800"
Oct 31 07:55:41.165218 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:41.165179    5829 generic.go:356] "Generic (PLEG): container finished" podID="d3c59a7a-9762-4473-953c-bb8ae815eac4" containerID="1a526e5b697ddba34dd02cd4ebd12aa8e6394456bb1c36d5ced00bfebe0d3f31" exitCode=0
Oct 31 07:55:41.165728 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:41.165261    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4" event={"ID":"d3c59a7a-9762-4473-953c-bb8ae815eac4","Type":"ContainerDied","Data":"1a526e5b697ddba34dd02cd4ebd12aa8e6394456bb1c36d5ced00bfebe0d3f31"}
Oct 31 07:55:42.307099 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.307066    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:42.438419 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.438314    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-openshift-ca\") pod \"d3c59a7a-9762-4473-953c-bb8ae815eac4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") "
Oct 31 07:55:42.438419 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.438402    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-xks-ca\") pod \"d3c59a7a-9762-4473-953c-bb8ae815eac4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") "
Oct 31 07:55:42.438656 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.438507    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c59a7a-9762-4473-953c-bb8ae815eac4-kube-api-access-5jvwn\" (UniqueName: \"kubernetes.io/projected/d3c59a7a-9762-4473-953c-bb8ae815eac4-kube-api-access-5jvwn\") pod \"d3c59a7a-9762-4473-953c-bb8ae815eac4\" (UID: \"d3c59a7a-9762-4473-953c-bb8ae815eac4\") "
Oct 31 07:55:42.438711 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.438683    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-openshift-ca" pod "d3c59a7a-9762-4473-953c-bb8ae815eac4" (UID: "d3c59a7a-9762-4473-953c-bb8ae815eac4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:55:42.438886 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.438866    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:42.440736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.440706    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-xks-ca" pod "d3c59a7a-9762-4473-953c-bb8ae815eac4" (UID: "d3c59a7a-9762-4473-953c-bb8ae815eac4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:55:42.440736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.440733    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c59a7a-9762-4473-953c-bb8ae815eac4-kube-api-access-5jvwn" pod "d3c59a7a-9762-4473-953c-bb8ae815eac4" (UID: "d3c59a7a-9762-4473-953c-bb8ae815eac4"). InnerVolumeSpecName "kube-api-access-5jvwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:55:42.525694 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.525649    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk" podUID="25954d97-3fc0-4399-a7de-9e843a41c549" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:55:42.525694 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.525686    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs" podUID="b1f47102-5e2c-4f62-8cfb-2adf62f1a474" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:55:42.540255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.540220    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c59a7a-9762-4473-953c-bb8ae815eac4-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:42.540255 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:42.540249    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jvwn\" (UniqueName: \"kubernetes.io/projected/d3c59a7a-9762-4473-953c-bb8ae815eac4-kube-api-access-5jvwn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:43.178445 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:43.178413    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"
Oct 31 07:55:43.178445 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:43.178427    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4" event={"ID":"d3c59a7a-9762-4473-953c-bb8ae815eac4","Type":"ContainerDied","Data":"ec7bd5300ad78c0a8bb8eae458822595fc43af74a40d2669b91d491c072faa64"}
Oct 31 07:55:43.178660 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:43.178481    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec7bd5300ad78c0a8bb8eae458822595fc43af74a40d2669b91d491c072faa64"
Oct 31 07:55:44.621708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.621676    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:55:44.637563 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.637538    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:55:44.764580 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.764548    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25954d97-3fc0-4399-a7de-9e843a41c549-kube-api-access-vnwjv\" (UniqueName: \"kubernetes.io/projected/25954d97-3fc0-4399-a7de-9e843a41c549-kube-api-access-vnwjv\") pod \"25954d97-3fc0-4399-a7de-9e843a41c549\" (UID: \"25954d97-3fc0-4399-a7de-9e843a41c549\") "
Oct 31 07:55:44.764772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.764629    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-plugins-config-volume\") pod \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\" (UID: \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\") "
Oct 31 07:55:44.764772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.764650    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-kube-api-access-m8hr5\" (UniqueName: \"kubernetes.io/projected/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-kube-api-access-m8hr5\") pod \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\" (UID: \"b1f47102-5e2c-4f62-8cfb-2adf62f1a474\") "
Oct 31 07:55:44.764772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.764712    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25954d97-3fc0-4399-a7de-9e843a41c549-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25954d97-3fc0-4399-a7de-9e843a41c549-plugins-config-volume\") pod \"25954d97-3fc0-4399-a7de-9e843a41c549\" (UID: \"25954d97-3fc0-4399-a7de-9e843a41c549\") "
Oct 31 07:55:44.765154 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.765122    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-plugins-config-volume" pod "b1f47102-5e2c-4f62-8cfb-2adf62f1a474" (UID: "b1f47102-5e2c-4f62-8cfb-2adf62f1a474"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:55:44.765352 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.765125    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25954d97-3fc0-4399-a7de-9e843a41c549-plugins-config-volume" pod "25954d97-3fc0-4399-a7de-9e843a41c549" (UID: "25954d97-3fc0-4399-a7de-9e843a41c549"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:55:44.766775 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.766749    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25954d97-3fc0-4399-a7de-9e843a41c549-kube-api-access-vnwjv" pod "25954d97-3fc0-4399-a7de-9e843a41c549" (UID: "25954d97-3fc0-4399-a7de-9e843a41c549"). InnerVolumeSpecName "kube-api-access-vnwjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:55:44.766874 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.766836    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-kube-api-access-m8hr5" pod "b1f47102-5e2c-4f62-8cfb-2adf62f1a474" (UID: "b1f47102-5e2c-4f62-8cfb-2adf62f1a474"). InnerVolumeSpecName "kube-api-access-m8hr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:55:44.866513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.866425    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnwjv\" (UniqueName: \"kubernetes.io/projected/25954d97-3fc0-4399-a7de-9e843a41c549-kube-api-access-vnwjv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:44.866513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.866456    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:44.866513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.866466    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8hr5\" (UniqueName: \"kubernetes.io/projected/b1f47102-5e2c-4f62-8cfb-2adf62f1a474-kube-api-access-m8hr5\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:44.866513 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:44.866477    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25954d97-3fc0-4399-a7de-9e843a41c549-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:55:45.190882 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.190842    5829 generic.go:356] "Generic (PLEG): container finished" podID="b1f47102-5e2c-4f62-8cfb-2adf62f1a474" containerID="728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685" exitCode=0
Oct 31 07:55:45.191094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.190898    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"
Oct 31 07:55:45.191094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.190925    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs" event={"ID":"b1f47102-5e2c-4f62-8cfb-2adf62f1a474","Type":"ContainerDied","Data":"728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685"}
Oct 31 07:55:45.191094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.190989    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs" event={"ID":"b1f47102-5e2c-4f62-8cfb-2adf62f1a474","Type":"ContainerDied","Data":"2179abcfafef5212f12282ba6154bcc0f2f600f47db8bec9ceae0f67394cca71"}
Oct 31 07:55:45.191094 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.191020    5829 scope.go:122] "RemoveContainer" containerID="728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685"
Oct 31 07:55:45.192437 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.192411    5829 generic.go:356] "Generic (PLEG): container finished" podID="25954d97-3fc0-4399-a7de-9e843a41c549" containerID="dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0" exitCode=0
Oct 31 07:55:45.192578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.192445    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk" event={"ID":"25954d97-3fc0-4399-a7de-9e843a41c549","Type":"ContainerDied","Data":"dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0"}
Oct 31 07:55:45.192578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.192501    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk" event={"ID":"25954d97-3fc0-4399-a7de-9e843a41c549","Type":"ContainerDied","Data":"d301e832fe1369b1249b1f7976507fcf19217955153fcde9a687998a7a528505"}
Oct 31 07:55:45.192578 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.192466    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"
Oct 31 07:55:45.202297 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.202265    5829 scope.go:122] "RemoveContainer" containerID="728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685"
Oct 31 07:55:45.202624 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:55:45.202602    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685\": container with ID starting with 728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685 not found: ID does not exist" containerID="728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685"
Oct 31 07:55:45.202708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.202633    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685"} err="failed to get container status \"728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685\": rpc error: code = NotFound desc = could not find container \"728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685\": container with ID starting with 728cf8419f4d055972b78e842a9e29dc89fbf1b024bf7b039639381f419fb685 not found: ID does not exist"
Oct 31 07:55:45.202708 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.202656    5829 scope.go:122] "RemoveContainer" containerID="dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0"
Oct 31 07:55:45.212979 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.212955    5829 scope.go:122] "RemoveContainer" containerID="dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0"
Oct 31 07:55:45.213380 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:55:45.213349    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0\": container with ID starting with dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0 not found: ID does not exist" containerID="dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0"
Oct 31 07:55:45.213463 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.213392    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0"} err="failed to get container status \"dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0\": rpc error: code = NotFound desc = could not find container \"dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0\": container with ID starting with dbf9e79bd0b5cd0ca9e7510aba86e5af9deca5ee37af1418abfe730202536ff0 not found: ID does not exist"
Oct 31 07:55:45.243338 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.243305    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"]
Oct 31 07:55:45.256133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.256103    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-009b0f-59678c4bb6gcjk"]
Oct 31 07:55:45.278905 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.278873    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"]
Oct 31 07:55:45.280549 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.280519    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-009b0f-56dddn5rhs"]
Oct 31 07:55:45.972723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.972688    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25954d97-3fc0-4399-a7de-9e843a41c549" path="/var/lib/kubelet/pods/25954d97-3fc0-4399-a7de-9e843a41c549/volumes"
Oct 31 07:55:45.973098 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:55:45.973076    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f47102-5e2c-4f62-8cfb-2adf62f1a474" path="/var/lib/kubelet/pods/b1f47102-5e2c-4f62-8cfb-2adf62f1a474/volumes"
Oct 31 07:55:46.510794 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:55:46.510756    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:56:16.510571 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:56:16.510481    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:56:17.025487 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:56:17.025445    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"]
Oct 31 07:56:17.038491 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:56:17.038460    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7chr9"]
Oct 31 07:56:17.973069 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:56:17.973030    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8050eec0-fbfe-4df1-bf56-9fc0f4c933f3" path="/var/lib/kubelet/pods/8050eec0-fbfe-4df1-bf56-9fc0f4c933f3/volumes"
Oct 31 07:56:46.510051 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:56:46.510010    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:57:16.512360 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:57:16.512315    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:57:18.074576 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:18.074538    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:57:30.533003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.532960    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"]
Oct 31 07:57:30.533509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.533471    5829 cpu_manager.go:414] "containerMap: removing container" podUID="25954d97-3fc0-4399-a7de-9e843a41c549" containerName="payload-processing"
Oct 31 07:57:30.533509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.533485    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25954d97-3fc0-4399-a7de-9e843a41c549" containerName="payload-processing"
Oct 31 07:57:30.533509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.533504    5829 cpu_manager.go:414] "containerMap: removing container" podUID="b1f47102-5e2c-4f62-8cfb-2adf62f1a474" containerName="payload-pre-processing"
Oct 31 07:57:30.533509 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.533509    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f47102-5e2c-4f62-8cfb-2adf62f1a474" containerName="payload-pre-processing"
Oct 31 07:57:30.533723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.533522    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d3c59a7a-9762-4473-953c-bb8ae815eac4" containerName="revoke-keys"
Oct 31 07:57:30.533723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.533527    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c59a7a-9762-4473-953c-bb8ae815eac4" containerName="revoke-keys"
Oct 31 07:57:30.533723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.533622    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="25954d97-3fc0-4399-a7de-9e843a41c549" containerName="payload-processing"
Oct 31 07:57:30.533723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.533633    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f47102-5e2c-4f62-8cfb-2adf62f1a474" containerName="payload-pre-processing"
Oct 31 07:57:30.533723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.533640    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c59a7a-9762-4473-953c-bb8ae815eac4" containerName="revoke-keys"
Oct 31 07:57:30.537023 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.537004    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.555174 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.555136    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"]
Oct 31 07:57:30.565029 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.564990    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:57:30.565216 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.565030    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:57:30.565216 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.565115    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:57:30.565560 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.565543    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:57:30.583736 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.583709    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-9xzhh\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.583870 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.583759    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqxm\" (UniqueName: \"kubernetes.io/projected/bb76d699-99f7-438d-b064-064d7dcb5f2c-kube-api-access-wrqxm\") pod \"maas-api-revoke-sub-d576e68b-9xzhh\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.583870 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.583780    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-9xzhh\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.684799 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.684763    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-9xzhh\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.684983 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.684818    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqxm\" (UniqueName: \"kubernetes.io/projected/bb76d699-99f7-438d-b064-064d7dcb5f2c-kube-api-access-wrqxm\") pod \"maas-api-revoke-sub-d576e68b-9xzhh\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.684983 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.684840    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-9xzhh\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.685492 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.685456    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-9xzhh\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.687195 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.687173    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-9xzhh\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.704630 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.704596    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqxm\" (UniqueName: \"kubernetes.io/projected/bb76d699-99f7-438d-b064-064d7dcb5f2c-kube-api-access-wrqxm\") pod \"maas-api-revoke-sub-d576e68b-9xzhh\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:30.850115 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:30.850022    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:31.149151 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:31.149123    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"]
Oct 31 07:57:31.151856 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:57:31.151820    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb76d699_99f7_438d_b064_064d7dcb5f2c.slice/crio-906d145e1fd5f1e5835b8fe6b3b8f1fc4aa22d58ca548833c46523df694b7e53 WatchSource:0}: Error finding container 906d145e1fd5f1e5835b8fe6b3b8f1fc4aa22d58ca548833c46523df694b7e53: Status 404 returned error can't find the container with id 906d145e1fd5f1e5835b8fe6b3b8f1fc4aa22d58ca548833c46523df694b7e53
Oct 31 07:57:31.813889 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:31.813790    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh" event={"ID":"bb76d699-99f7-438d-b064-064d7dcb5f2c","Type":"ContainerStarted","Data":"ce8e1f13bde12d7855622cc009409bda5f9c8605e1fdf6458bd5bd003ab0345f"}
Oct 31 07:57:31.813889 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:31.813855    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh" event={"ID":"bb76d699-99f7-438d-b064-064d7dcb5f2c","Type":"ContainerStarted","Data":"906d145e1fd5f1e5835b8fe6b3b8f1fc4aa22d58ca548833c46523df694b7e53"}
Oct 31 07:57:31.858253 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:31.858195    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh" podStartSLOduration=1.858180635 podStartE2EDuration="1.858180635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:57:31.8523853 +0000 UTC m=+1715.991545885" watchObservedRunningTime="2026-10-31 07:57:31.858180635 +0000 UTC m=+1715.997341214"
Oct 31 07:57:32.820435 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:32.820397    5829 generic.go:356] "Generic (PLEG): container finished" podID="bb76d699-99f7-438d-b064-064d7dcb5f2c" containerID="ce8e1f13bde12d7855622cc009409bda5f9c8605e1fdf6458bd5bd003ab0345f" exitCode=0
Oct 31 07:57:32.820856 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:32.820477    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh" event={"ID":"bb76d699-99f7-438d-b064-064d7dcb5f2c","Type":"ContainerDied","Data":"ce8e1f13bde12d7855622cc009409bda5f9c8605e1fdf6458bd5bd003ab0345f"}
Oct 31 07:57:33.968455 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:33.968425    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:34.022615 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.022577    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb76d699-99f7-438d-b064-064d7dcb5f2c-kube-api-access-wrqxm\" (UniqueName: \"kubernetes.io/projected/bb76d699-99f7-438d-b064-064d7dcb5f2c-kube-api-access-wrqxm\") pod \"bb76d699-99f7-438d-b064-064d7dcb5f2c\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") "
Oct 31 07:57:34.022809 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.022698    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-openshift-ca\") pod \"bb76d699-99f7-438d-b064-064d7dcb5f2c\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") "
Oct 31 07:57:34.022877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.022856    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-xks-ca\") pod \"bb76d699-99f7-438d-b064-064d7dcb5f2c\" (UID: \"bb76d699-99f7-438d-b064-064d7dcb5f2c\") "
Oct 31 07:57:34.023101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.023072    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-openshift-ca" pod "bb76d699-99f7-438d-b064-064d7dcb5f2c" (UID: "bb76d699-99f7-438d-b064-064d7dcb5f2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:57:34.024753 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.024722    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb76d699-99f7-438d-b064-064d7dcb5f2c-kube-api-access-wrqxm" pod "bb76d699-99f7-438d-b064-064d7dcb5f2c" (UID: "bb76d699-99f7-438d-b064-064d7dcb5f2c"). InnerVolumeSpecName "kube-api-access-wrqxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:57:34.024853 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.024779    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-xks-ca" pod "bb76d699-99f7-438d-b064-064d7dcb5f2c" (UID: "bb76d699-99f7-438d-b064-064d7dcb5f2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:57:34.123901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.123815    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:57:34.123901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.123849    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqxm\" (UniqueName: \"kubernetes.io/projected/bb76d699-99f7-438d-b064-064d7dcb5f2c-kube-api-access-wrqxm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:57:34.123901 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.123859    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb76d699-99f7-438d-b064-064d7dcb5f2c-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:57:34.834086 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.834058    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"
Oct 31 07:57:34.834276 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.834101    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh" event={"ID":"bb76d699-99f7-438d-b064-064d7dcb5f2c","Type":"ContainerDied","Data":"906d145e1fd5f1e5835b8fe6b3b8f1fc4aa22d58ca548833c46523df694b7e53"}
Oct 31 07:57:34.834276 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:34.834150    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="906d145e1fd5f1e5835b8fe6b3b8f1fc4aa22d58ca548833c46523df694b7e53"
Oct 31 07:57:41.756748 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:41.756657    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:57:46.511264 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:57:46.511221    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:57:47.026450 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:47.026418    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"]
Oct 31 07:57:47.039209 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:47.039165    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-244lv"]
Oct 31 07:57:47.973176 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:47.973126    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5552e9-740e-4db9-a717-25c938fa6a37" path="/var/lib/kubelet/pods/8e5552e9-740e-4db9-a717-25c938fa6a37/volumes"
Oct 31 07:57:57.032271 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:57.032236    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"]
Oct 31 07:57:57.045113 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:57.045079    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-r5xh6"]
Oct 31 07:57:57.972855 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:57.972816    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a" path="/var/lib/kubelet/pods/7ac7ea7f-fa0e-4922-8e87-2f239cc5be5a/volumes"
Oct 31 07:57:58.608202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:58.608167    5829 scope.go:122] "RemoveContainer" containerID="e91c31cc9ac45d0dd64344d6fe227cef73049f09ae20bff1f257cc4e14475362"
Oct 31 07:57:58.617115 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:58.617097    5829 scope.go:122] "RemoveContainer" containerID="89a14041a407c6817dccef9f258df5b83a347eb084b297b7cd754a4646b49a70"
Oct 31 07:57:58.625435 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:57:58.625416    5829 scope.go:122] "RemoveContainer" containerID="b1c4410cf1a0b6317736e567be8888ccecbe21a0d569d89d7d378c1e4c343c5f"
Oct 31 07:58:13.364373 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:13.364337    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:58:16.511738 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:58:16.511689    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:58:18.566783 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.566746    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"]
Oct 31 07:58:18.567234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.567221    5829 cpu_manager.go:414] "containerMap: removing container" podUID="bb76d699-99f7-438d-b064-064d7dcb5f2c" containerName="revoke-keys"
Oct 31 07:58:18.567321 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.567236    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb76d699-99f7-438d-b064-064d7dcb5f2c" containerName="revoke-keys"
Oct 31 07:58:18.567370 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.567324    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb76d699-99f7-438d-b064-064d7dcb5f2c" containerName="revoke-keys"
Oct 31 07:58:18.570503 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.570485    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.588194 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.588161    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"]
Oct 31 07:58:18.598234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.598205    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:58:18.598234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.598229    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:58:18.598234 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.598208    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:58:18.599376 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.599356    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:58:18.613626 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.613594    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfpph\" (UniqueName: \"kubernetes.io/projected/53af56f8-9ea8-4b4e-b703-80b39be83126-kube-api-access-lfpph\") pod \"maas-api-revoke-sub-d2bc93bc-g7tm9\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.613787 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.613717    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g7tm9\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.613787 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.613768    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g7tm9\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.714624 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.714587    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfpph\" (UniqueName: \"kubernetes.io/projected/53af56f8-9ea8-4b4e-b703-80b39be83126-kube-api-access-lfpph\") pod \"maas-api-revoke-sub-d2bc93bc-g7tm9\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.714806 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.714723    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g7tm9\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.714806 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.714790    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g7tm9\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.715534 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.715500    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g7tm9\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.717233 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.717213    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-g7tm9\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.734533 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.734502    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfpph\" (UniqueName: \"kubernetes.io/projected/53af56f8-9ea8-4b4e-b703-80b39be83126-kube-api-access-lfpph\") pod \"maas-api-revoke-sub-d2bc93bc-g7tm9\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:18.884034 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:18.883948    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:19.184810 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:19.184784    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"]
Oct 31 07:58:19.187213 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:58:19.187181    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53af56f8_9ea8_4b4e_b703_80b39be83126.slice/crio-fd8746c84b170d42e45757a52e6bd0172763df3f8f093319b0270238284cb023 WatchSource:0}: Error finding container fd8746c84b170d42e45757a52e6bd0172763df3f8f093319b0270238284cb023: Status 404 returned error can't find the container with id fd8746c84b170d42e45757a52e6bd0172763df3f8f093319b0270238284cb023
Oct 31 07:58:20.107843 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:20.107807    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9" event={"ID":"53af56f8-9ea8-4b4e-b703-80b39be83126","Type":"ContainerStarted","Data":"66ae4b182d51d8c2254d941503a14d9b54afaa9211bba64331bb0ab2ecaa0ead"}
Oct 31 07:58:20.108254 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:20.107873    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9" event={"ID":"53af56f8-9ea8-4b4e-b703-80b39be83126","Type":"ContainerStarted","Data":"fd8746c84b170d42e45757a52e6bd0172763df3f8f093319b0270238284cb023"}
Oct 31 07:58:20.147169 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:20.147052    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9" podStartSLOduration=2.147039499 podStartE2EDuration="2.147039499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:58:20.146595314 +0000 UTC m=+1764.285755881" watchObservedRunningTime="2026-10-31 07:58:20.147039499 +0000 UTC m=+1764.286200090"
Oct 31 07:58:21.114504 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:21.114418    5829 generic.go:356] "Generic (PLEG): container finished" podID="53af56f8-9ea8-4b4e-b703-80b39be83126" containerID="66ae4b182d51d8c2254d941503a14d9b54afaa9211bba64331bb0ab2ecaa0ead" exitCode=0
Oct 31 07:58:21.114998 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:21.114596    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9" event={"ID":"53af56f8-9ea8-4b4e-b703-80b39be83126","Type":"ContainerDied","Data":"66ae4b182d51d8c2254d941503a14d9b54afaa9211bba64331bb0ab2ecaa0ead"}
Oct 31 07:58:22.263933 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.263906    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:22.346597 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.346562    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-xks-ca\") pod \"53af56f8-9ea8-4b4e-b703-80b39be83126\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") "
Oct 31 07:58:22.346772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.346634    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53af56f8-9ea8-4b4e-b703-80b39be83126-kube-api-access-lfpph\" (UniqueName: \"kubernetes.io/projected/53af56f8-9ea8-4b4e-b703-80b39be83126-kube-api-access-lfpph\") pod \"53af56f8-9ea8-4b4e-b703-80b39be83126\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") "
Oct 31 07:58:22.346772 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.346682    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-openshift-ca\") pod \"53af56f8-9ea8-4b4e-b703-80b39be83126\" (UID: \"53af56f8-9ea8-4b4e-b703-80b39be83126\") "
Oct 31 07:58:22.347069 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.347045    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-openshift-ca" pod "53af56f8-9ea8-4b4e-b703-80b39be83126" (UID: "53af56f8-9ea8-4b4e-b703-80b39be83126"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:58:22.348781 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.348755    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-xks-ca" pod "53af56f8-9ea8-4b4e-b703-80b39be83126" (UID: "53af56f8-9ea8-4b4e-b703-80b39be83126"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:58:22.348906 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.348799    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53af56f8-9ea8-4b4e-b703-80b39be83126-kube-api-access-lfpph" pod "53af56f8-9ea8-4b4e-b703-80b39be83126" (UID: "53af56f8-9ea8-4b4e-b703-80b39be83126"). InnerVolumeSpecName "kube-api-access-lfpph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:58:22.447757 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.447713    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:58:22.447757 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.447754    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfpph\" (UniqueName: \"kubernetes.io/projected/53af56f8-9ea8-4b4e-b703-80b39be83126-kube-api-access-lfpph\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:58:22.447757 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:22.447764    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53af56f8-9ea8-4b4e-b703-80b39be83126-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:58:23.126734 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:23.126701    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"
Oct 31 07:58:23.126928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:23.126738    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9" event={"ID":"53af56f8-9ea8-4b4e-b703-80b39be83126","Type":"ContainerDied","Data":"fd8746c84b170d42e45757a52e6bd0172763df3f8f093319b0270238284cb023"}
Oct 31 07:58:23.126928 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:23.126794    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd8746c84b170d42e45757a52e6bd0172763df3f8f093319b0270238284cb023"
Oct 31 07:58:29.781747 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:29.781710    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:58:35.818529 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.818491    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"]
Oct 31 07:58:35.818936 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.818924    5829 cpu_manager.go:414] "containerMap: removing container" podUID="53af56f8-9ea8-4b4e-b703-80b39be83126" containerName="revoke-keys"
Oct 31 07:58:35.818980 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.818937    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53af56f8-9ea8-4b4e-b703-80b39be83126" containerName="revoke-keys"
Oct 31 07:58:35.819050 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.819041    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="53af56f8-9ea8-4b4e-b703-80b39be83126" containerName="revoke-keys"
Oct 31 07:58:35.822745 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.822722    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:35.838939 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.838909    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-11a81e\""
Oct 31 07:58:35.841213 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.841182    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"]
Oct 31 07:58:35.884396 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.884231    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xwm\" (UniqueName: \"kubernetes.io/projected/cd2db893-738f-4f0a-9313-7227070e6a70-kube-api-access-b2xwm\") pod \"payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd\" (UID: \"cd2db893-738f-4f0a-9313-7227070e6a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:35.884579 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.884528    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2db893-738f-4f0a-9313-7227070e6a70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd\" (UID: \"cd2db893-738f-4f0a-9313-7227070e6a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:35.898904 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.898868    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-32753b98-dockercfg-gvtjl\""
Oct 31 07:58:35.906673 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.906641    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"]
Oct 31 07:58:35.910817 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.910780    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:35.928575 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.928543    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"]
Oct 31 07:58:35.986020 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.985985    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqzvt\" (UniqueName: \"kubernetes.io/projected/16b70b2b-fe40-4196-a127-fcbf0c38ca92-kube-api-access-jqzvt\") pod \"payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v\" (UID: \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:35.986202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.986048    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2db893-738f-4f0a-9313-7227070e6a70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd\" (UID: \"cd2db893-738f-4f0a-9313-7227070e6a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:35.986202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.986148    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b70b2b-fe40-4196-a127-fcbf0c38ca92-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v\" (UID: \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:35.986202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.986189    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xwm\" (UniqueName: \"kubernetes.io/projected/cd2db893-738f-4f0a-9313-7227070e6a70-kube-api-access-b2xwm\") pod \"payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd\" (UID: \"cd2db893-738f-4f0a-9313-7227070e6a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:35.986855 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:35.986829    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2db893-738f-4f0a-9313-7227070e6a70-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd\" (UID: \"cd2db893-738f-4f0a-9313-7227070e6a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:36.006361 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:36.006316    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xwm\" (UniqueName: \"kubernetes.io/projected/cd2db893-738f-4f0a-9313-7227070e6a70-kube-api-access-b2xwm\") pod \"payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd\" (UID: \"cd2db893-738f-4f0a-9313-7227070e6a70\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:36.087840 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:36.087805    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b70b2b-fe40-4196-a127-fcbf0c38ca92-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v\" (UID: \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:36.088043 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:36.087903    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqzvt\" (UniqueName: \"kubernetes.io/projected/16b70b2b-fe40-4196-a127-fcbf0c38ca92-kube-api-access-jqzvt\") pod \"payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v\" (UID: \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:36.088814 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:36.088785    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b70b2b-fe40-4196-a127-fcbf0c38ca92-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v\" (UID: \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:36.111854 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:36.111825    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqzvt\" (UniqueName: \"kubernetes.io/projected/16b70b2b-fe40-4196-a127-fcbf0c38ca92-kube-api-access-jqzvt\") pod \"payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v\" (UID: \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:36.136806 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:36.136714    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:36.224568 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:36.224522    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:36.439221 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:36.439183    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"]
Oct 31 07:58:36.441864 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:58:36.441830    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd2db893_738f_4f0a_9313_7227070e6a70.slice/crio-2f00268e0a6be8fdefc9c4aab039935b5ae04769c8fdb720cfc172c81d78654e WatchSource:0}: Error finding container 2f00268e0a6be8fdefc9c4aab039935b5ae04769c8fdb720cfc172c81d78654e: Status 404 returned error can't find the container with id 2f00268e0a6be8fdefc9c4aab039935b5ae04769c8fdb720cfc172c81d78654e
Oct 31 07:58:36.529915 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:36.529889    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"]
Oct 31 07:58:36.532142 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:58:36.532111    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b70b2b_fe40_4196_a127_fcbf0c38ca92.slice/crio-5d4998d604cc55518be0fa3ecc52f4233c25e5ab093346acca59d53e5c37a33b WatchSource:0}: Error finding container 5d4998d604cc55518be0fa3ecc52f4233c25e5ab093346acca59d53e5c37a33b: Status 404 returned error can't find the container with id 5d4998d604cc55518be0fa3ecc52f4233c25e5ab093346acca59d53e5c37a33b
Oct 31 07:58:37.207794 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:37.207760    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd" event={"ID":"cd2db893-738f-4f0a-9313-7227070e6a70","Type":"ContainerStarted","Data":"e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9"}
Oct 31 07:58:37.208224 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:37.207813    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd" event={"ID":"cd2db893-738f-4f0a-9313-7227070e6a70","Type":"ContainerStarted","Data":"2f00268e0a6be8fdefc9c4aab039935b5ae04769c8fdb720cfc172c81d78654e"}
Oct 31 07:58:37.208224 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:37.207939    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:37.209371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:37.209345    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v" event={"ID":"16b70b2b-fe40-4196-a127-fcbf0c38ca92","Type":"ContainerStarted","Data":"a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58"}
Oct 31 07:58:37.209489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:37.209386    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v" event={"ID":"16b70b2b-fe40-4196-a127-fcbf0c38ca92","Type":"ContainerStarted","Data":"5d4998d604cc55518be0fa3ecc52f4233c25e5ab093346acca59d53e5c37a33b"}
Oct 31 07:58:37.209819 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:37.209804    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:37.249444 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:37.249384    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd" podStartSLOduration=2.249369148 podStartE2EDuration="2.249369148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:58:37.247840583 +0000 UTC m=+1781.387001170" watchObservedRunningTime="2026-10-31 07:58:37.249369148 +0000 UTC m=+1781.388529787"
Oct 31 07:58:37.291103 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:37.291047    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v" podStartSLOduration=2.291030648 podStartE2EDuration="2.291030648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:58:37.289810525 +0000 UTC m=+1781.428971113" watchObservedRunningTime="2026-10-31 07:58:37.291030648 +0000 UTC m=+1781.430191235"
Oct 31 07:58:38.884489 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:38.884438    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:58:46.511529 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:58:46.511488    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:58:48.217371 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:48.217340    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 07:58:49.223650 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:49.223621    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 07:58:54.577721 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:54.577686    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:58:58.157464 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:58.157432    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:58:58.159393 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:58.159372    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 07:58:58.665333 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:58.665271    5829 scope.go:122] "RemoveContainer" containerID="9f8eddc956021c2bd30d746204b83ba50efd8b7029b7750c221cce8253aebf47"
Oct 31 07:58:58.674320 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:58:58.674273    5829 scope.go:122] "RemoveContainer" containerID="2d275a51ba73718d25fa0451762f411206115b8ffad50e7c8031f53809e66752"
Oct 31 07:59:08.863091 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:08.863002    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:59:16.511639 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:59:16.511598    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:59:21.135448 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.135412    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"]
Oct 31 07:59:21.138978 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.138960    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.156408 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.156379    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"]
Oct 31 07:59:21.167247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.167156    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:59:21.167247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.167192    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:59:21.167247 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.167223    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:59:21.167493 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.167344    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:59:21.243718 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.243676    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef47abfe-h4hhq\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.243718 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.243724    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdzm\" (UniqueName: \"kubernetes.io/projected/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-kube-api-access-gjdzm\") pod \"maas-api-revoke-sub-ef47abfe-h4hhq\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.243933 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.243843    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef47abfe-h4hhq\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.344705 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.344666    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef47abfe-h4hhq\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.344900 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.344785    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef47abfe-h4hhq\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.344900 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.344811    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdzm\" (UniqueName: \"kubernetes.io/projected/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-kube-api-access-gjdzm\") pod \"maas-api-revoke-sub-ef47abfe-h4hhq\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.345547 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.345527    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef47abfe-h4hhq\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.347038 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.347021    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef47abfe-h4hhq\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.364548 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.364522    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdzm\" (UniqueName: \"kubernetes.io/projected/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-kube-api-access-gjdzm\") pod \"maas-api-revoke-sub-ef47abfe-h4hhq\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.452620 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.452581    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:21.754449 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:21.754425    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"]
Oct 31 07:59:21.757323 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:59:21.757296    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc28bb60_e2f2_4465_8dfb_b7e65b9fdb21.slice/crio-dacd2e17276a2b06d536f6a94adeafd803654b2dd0e625ebf060c4da8fec50f7 WatchSource:0}: Error finding container dacd2e17276a2b06d536f6a94adeafd803654b2dd0e625ebf060c4da8fec50f7: Status 404 returned error can't find the container with id dacd2e17276a2b06d536f6a94adeafd803654b2dd0e625ebf060c4da8fec50f7
Oct 31 07:59:22.468105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:22.468064    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq" event={"ID":"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21","Type":"ContainerStarted","Data":"a9892d699418687297dc5b38c0f3ab5dd3d7b9fea70e8b07462f5bac0931e9a2"}
Oct 31 07:59:22.468647 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:22.468117    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq" event={"ID":"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21","Type":"ContainerStarted","Data":"dacd2e17276a2b06d536f6a94adeafd803654b2dd0e625ebf060c4da8fec50f7"}
Oct 31 07:59:22.511408 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:22.511351    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq" podStartSLOduration=1.5113373399999999 podStartE2EDuration="1.51133734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:59:22.510599429 +0000 UTC m=+1826.649760019" watchObservedRunningTime="2026-10-31 07:59:22.51133734 +0000 UTC m=+1826.650497928"
Oct 31 07:59:23.475119 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:23.475077    5829 generic.go:356] "Generic (PLEG): container finished" podID="fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21" containerID="a9892d699418687297dc5b38c0f3ab5dd3d7b9fea70e8b07462f5bac0931e9a2" exitCode=0
Oct 31 07:59:23.475559 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:23.475129    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq" event={"ID":"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21","Type":"ContainerDied","Data":"a9892d699418687297dc5b38c0f3ab5dd3d7b9fea70e8b07462f5bac0931e9a2"}
Oct 31 07:59:24.630679 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.630647    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:24.777105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.777018    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-kube-api-access-gjdzm\" (UniqueName: \"kubernetes.io/projected/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-kube-api-access-gjdzm\") pod \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") "
Oct 31 07:59:24.777105 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.777094    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-openshift-ca\") pod \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") "
Oct 31 07:59:24.777345 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.777167    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-xks-ca\") pod \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\" (UID: \"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21\") "
Oct 31 07:59:24.777500 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.777473    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-openshift-ca" pod "fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21" (UID: "fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:59:24.779175 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.779135    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-kube-api-access-gjdzm" pod "fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21" (UID: "fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21"). InnerVolumeSpecName "kube-api-access-gjdzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:59:24.779300 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.779240    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-xks-ca" pod "fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21" (UID: "fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:59:24.878250 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.878207    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:59:24.878250 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.878250    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdzm\" (UniqueName: \"kubernetes.io/projected/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-kube-api-access-gjdzm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:59:24.878507 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:24.878267    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:59:25.487747 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:25.487704    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq" event={"ID":"fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21","Type":"ContainerDied","Data":"dacd2e17276a2b06d536f6a94adeafd803654b2dd0e625ebf060c4da8fec50f7"}
Oct 31 07:59:25.487935 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:25.487764    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dacd2e17276a2b06d536f6a94adeafd803654b2dd0e625ebf060c4da8fec50f7"
Oct 31 07:59:25.487935 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:25.487717    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"
Oct 31 07:59:32.230623 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.230586    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"]
Oct 31 07:59:32.231049 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.231036    5829 cpu_manager.go:414] "containerMap: removing container" podUID="fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21" containerName="revoke-keys"
Oct 31 07:59:32.231097 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.231050    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21" containerName="revoke-keys"
Oct 31 07:59:32.231162 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.231152    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21" containerName="revoke-keys"
Oct 31 07:59:32.234553 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.234528    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.251668 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.251639    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"]
Oct 31 07:59:32.262012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.261976    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 07:59:32.262012 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.261988    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 07:59:32.263077 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.263057    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 07:59:32.263077 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.263075    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 07:59:32.358605 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.358573    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5hp\" (UniqueName: \"kubernetes.io/projected/2d38fafd-5f48-486e-99c2-ed0812795d79-kube-api-access-8c5hp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.358813 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.358783    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.358877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.358848    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.459877 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.459834    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5hp\" (UniqueName: \"kubernetes.io/projected/2d38fafd-5f48-486e-99c2-ed0812795d79-kube-api-access-8c5hp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.460095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.459992    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.460095 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.460038    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.460738 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.460713    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.462444 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.462422    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.479101 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.479073    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5hp\" (UniqueName: \"kubernetes.io/projected/2d38fafd-5f48-486e-99c2-ed0812795d79-kube-api-access-8c5hp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.547548 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.547450    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:32.674502 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.674465    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:59:32.844523 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:32.844498    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"]
Oct 31 07:59:32.846562 ip-10-0-140-236 kubenswrapper[5829]: W1031 07:59:32.846535    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d38fafd_5f48_486e_99c2_ed0812795d79.slice/crio-505462a5a375ecc32d3d7b667894a46dbe5951a53cecf8a25a03e3f6784529e8 WatchSource:0}: Error finding container 505462a5a375ecc32d3d7b667894a46dbe5951a53cecf8a25a03e3f6784529e8: Status 404 returned error can't find the container with id 505462a5a375ecc32d3d7b667894a46dbe5951a53cecf8a25a03e3f6784529e8
Oct 31 07:59:33.535474 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:33.535437    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7" event={"ID":"2d38fafd-5f48-486e-99c2-ed0812795d79","Type":"ContainerStarted","Data":"f4f7b904fa413486dc690f9bcd11352fc4fa82258ae6cf87d84764764a02a0e0"}
Oct 31 07:59:33.535906 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:33.535490    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7" event={"ID":"2d38fafd-5f48-486e-99c2-ed0812795d79","Type":"ContainerStarted","Data":"505462a5a375ecc32d3d7b667894a46dbe5951a53cecf8a25a03e3f6784529e8"}
Oct 31 07:59:33.574133 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:33.574076    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7" podStartSLOduration=1.57405963 podStartE2EDuration="1.57405963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:59:33.573652816 +0000 UTC m=+1837.712813405" watchObservedRunningTime="2026-10-31 07:59:33.57405963 +0000 UTC m=+1837.713220218"
Oct 31 07:59:34.542531 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:34.542493    5829 generic.go:356] "Generic (PLEG): container finished" podID="2d38fafd-5f48-486e-99c2-ed0812795d79" containerID="f4f7b904fa413486dc690f9bcd11352fc4fa82258ae6cf87d84764764a02a0e0" exitCode=0
Oct 31 07:59:34.542912 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:34.542577    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7" event={"ID":"2d38fafd-5f48-486e-99c2-ed0812795d79","Type":"ContainerDied","Data":"f4f7b904fa413486dc690f9bcd11352fc4fa82258ae6cf87d84764764a02a0e0"}
Oct 31 07:59:35.692246 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.692218    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:35.796202 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.796166    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-openshift-ca\") pod \"2d38fafd-5f48-486e-99c2-ed0812795d79\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") "
Oct 31 07:59:35.796419 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.796317    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d38fafd-5f48-486e-99c2-ed0812795d79-kube-api-access-8c5hp\" (UniqueName: \"kubernetes.io/projected/2d38fafd-5f48-486e-99c2-ed0812795d79-kube-api-access-8c5hp\") pod \"2d38fafd-5f48-486e-99c2-ed0812795d79\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") "
Oct 31 07:59:35.796419 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.796371    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-xks-ca\") pod \"2d38fafd-5f48-486e-99c2-ed0812795d79\" (UID: \"2d38fafd-5f48-486e-99c2-ed0812795d79\") "
Oct 31 07:59:35.796661 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.796629    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-openshift-ca" pod "2d38fafd-5f48-486e-99c2-ed0812795d79" (UID: "2d38fafd-5f48-486e-99c2-ed0812795d79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 07:59:35.796785 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.796770    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:59:35.798568 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.798496    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d38fafd-5f48-486e-99c2-ed0812795d79-kube-api-access-8c5hp" pod "2d38fafd-5f48-486e-99c2-ed0812795d79" (UID: "2d38fafd-5f48-486e-99c2-ed0812795d79"). InnerVolumeSpecName "kube-api-access-8c5hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 07:59:35.798568 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.798517    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-xks-ca" pod "2d38fafd-5f48-486e-99c2-ed0812795d79" (UID: "2d38fafd-5f48-486e-99c2-ed0812795d79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 07:59:35.897863 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.897822    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5hp\" (UniqueName: \"kubernetes.io/projected/2d38fafd-5f48-486e-99c2-ed0812795d79-kube-api-access-8c5hp\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:59:35.897863 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:35.897861    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d38fafd-5f48-486e-99c2-ed0812795d79-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 07:59:36.555298 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:36.555246    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"
Oct 31 07:59:36.555483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:36.555302    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7" event={"ID":"2d38fafd-5f48-486e-99c2-ed0812795d79","Type":"ContainerDied","Data":"505462a5a375ecc32d3d7b667894a46dbe5951a53cecf8a25a03e3f6784529e8"}
Oct 31 07:59:36.555483 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:36.555386    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505462a5a375ecc32d3d7b667894a46dbe5951a53cecf8a25a03e3f6784529e8"
Oct 31 07:59:42.629374 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:42.629336    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"]
Oct 31 07:59:42.659871 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:42.659834    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"]
Oct 31 07:59:45.155163 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:45.155121    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"]
Oct 31 07:59:45.168003 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:45.167964    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-11a81e-pw4w7"]
Oct 31 07:59:45.973581 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:45.973545    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d38fafd-5f48-486e-99c2-ed0812795d79" path="/var/lib/kubelet/pods/2d38fafd-5f48-486e-99c2-ed0812795d79/volumes"
Oct 31 07:59:46.511407 ip-10-0-140-236 kubenswrapper[5829]: E1031 07:59:46.511370    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 07:59:46.869168 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:46.869068    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 07:59:47.630723 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:47.630649    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v" podUID="16b70b2b-fe40-4196-a127-fcbf0c38ca92" containerName="payload-processing" containerID="cri-o://a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58" gracePeriod=25
Oct 31 07:59:47.660421 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:47.660348    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd" podUID="cd2db893-738f-4f0a-9313-7227070e6a70" containerName="payload-pre-processing" containerID="cri-o://e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9" gracePeriod=25
Oct 31 07:59:48.216150 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:48.216103    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd" podUID="cd2db893-738f-4f0a-9313-7227070e6a70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:59:49.222521 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:49.222478    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v" podUID="16b70b2b-fe40-4196-a127-fcbf0c38ca92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:59:51.023608 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:51.023571    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"]
Oct 31 07:59:51.036622 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:51.036586    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ab697c-jlkkn"]
Oct 31 07:59:51.973330 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:51.973270    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d576fc9-635e-40bb-be85-6d0dc1955730" path="/var/lib/kubelet/pods/9d576fc9-635e-40bb-be85-6d0dc1955730/volumes"
Oct 31 07:59:58.216330 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:58.216261    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd" podUID="cd2db893-738f-4f0a-9313-7227070e6a70" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 07:59:58.699937 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:58.699902    5829 scope.go:122] "RemoveContainer" containerID="b4168c65863279a5ab4f76ddc1794c2abbc46e10d7abe1d09cb27238d4bfa1f7"
Oct 31 07:59:59.222071 ip-10-0-140-236 kubenswrapper[5829]: I1031 07:59:59.222032    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v" podUID="16b70b2b-fe40-4196-a127-fcbf0c38ca92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 08:00:00.141690 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.141653    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"]
Oct 31 08:00:00.142144 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.142132    5829 cpu_manager.go:414] "containerMap: removing container" podUID="2d38fafd-5f48-486e-99c2-ed0812795d79" containerName="revoke-keys"
Oct 31 08:00:00.142202 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.142146    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d38fafd-5f48-486e-99c2-ed0812795d79" containerName="revoke-keys"
Oct 31 08:00:00.142240 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.142210    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d38fafd-5f48-486e-99c2-ed0812795d79" containerName="revoke-keys"
Oct 31 08:00:00.146774 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.146757    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:00.163368 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.163339    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"]
Oct 31 08:00:00.173504 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.173481    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:00:00.173634 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.173509    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:00:00.173634 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.173564    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:00:00.256981 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.256947    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczfn\" (UniqueName: \"kubernetes.io/projected/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-kube-api-access-dczfn\") pod \"maas-api-key-cleanup-29890560-vjqrf\" (UID: \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:00.257427 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.257086    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890560-vjqrf\" (UID: \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:00.357643 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.357608    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczfn\" (UniqueName: \"kubernetes.io/projected/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-kube-api-access-dczfn\") pod \"maas-api-key-cleanup-29890560-vjqrf\" (UID: \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:00.357852 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.357704    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890560-vjqrf\" (UID: \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:00.358506 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.358477    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890560-vjqrf\" (UID: \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:00.377468 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.377435    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczfn\" (UniqueName: \"kubernetes.io/projected/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-kube-api-access-dczfn\") pod \"maas-api-key-cleanup-29890560-vjqrf\" (UID: \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:00.459795 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.459706    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:00.756843 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:00.756744    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"]
Oct 31 08:00:00.759460 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:00:00.759431    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7544fbc_ef1f_458b_a5e9_e588ca8cf34d.slice/crio-36beea4cb4299d29af14430e65ced8bbc6cb6b133f10f1c32f1702129e3bf441 WatchSource:0}: Error finding container 36beea4cb4299d29af14430e65ced8bbc6cb6b133f10f1c32f1702129e3bf441: Status 404 returned error can't find the container with id 36beea4cb4299d29af14430e65ced8bbc6cb6b133f10f1c32f1702129e3bf441
Oct 31 08:00:01.700681 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:01.700644    5829 generic.go:356] "Generic (PLEG): container finished" podID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerID="bc6bd3705e2e38858ad0b6303be778c400319e94195622ecd9787401b75e839f" exitCode=60
Oct 31 08:00:01.701115 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:01.700729    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" event={"ID":"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d","Type":"ContainerDied","Data":"bc6bd3705e2e38858ad0b6303be778c400319e94195622ecd9787401b75e839f"}
Oct 31 08:00:01.701115 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:01.700800    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" event={"ID":"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d","Type":"ContainerStarted","Data":"36beea4cb4299d29af14430e65ced8bbc6cb6b133f10f1c32f1702129e3bf441"}
Oct 31 08:00:01.701115 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:01.700969    5829 scope.go:122] "RemoveContainer" containerID="bc6bd3705e2e38858ad0b6303be778c400319e94195622ecd9787401b75e839f"
Oct 31 08:00:02.708829 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:02.708799    5829 generic.go:356] "Generic (PLEG): container finished" podID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerID="afc122284581539f5de193936dddef2f425d67aafe5071313113a62fa93a1e59" exitCode=60
Oct 31 08:00:02.709245 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:02.708893    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" event={"ID":"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d","Type":"ContainerDied","Data":"afc122284581539f5de193936dddef2f425d67aafe5071313113a62fa93a1e59"}
Oct 31 08:00:02.709245 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:02.708955    5829 scope.go:122] "RemoveContainer" containerID="bc6bd3705e2e38858ad0b6303be778c400319e94195622ecd9787401b75e839f"
Oct 31 08:00:02.709245 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:02.709155    5829 scope.go:122] "RemoveContainer" containerID="afc122284581539f5de193936dddef2f425d67aafe5071313113a62fa93a1e59"
Oct 31 08:00:02.709460 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:00:02.709441    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890560-vjqrf_odh-ai-gateway-infra(e7544fbc-ef1f-458b-a5e9-e588ca8cf34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d"
Oct 31 08:00:03.715790 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:03.715758    5829 scope.go:122] "RemoveContainer" containerID="afc122284581539f5de193936dddef2f425d67aafe5071313113a62fa93a1e59"
Oct 31 08:00:03.716195 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:00:03.716024    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890560-vjqrf_odh-ai-gateway-infra(e7544fbc-ef1f-458b-a5e9-e588ca8cf34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d"
Oct 31 08:00:07.925403 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:07.925369    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 08:00:07.949738 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:07.949710    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 08:00:08.038544 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.038512    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b70b2b-fe40-4196-a127-fcbf0c38ca92-kube-api-access-jqzvt\" (UniqueName: \"kubernetes.io/projected/16b70b2b-fe40-4196-a127-fcbf0c38ca92-kube-api-access-jqzvt\") pod \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\" (UID: \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\") "
Oct 31 08:00:08.038748 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.038608    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b70b2b-fe40-4196-a127-fcbf0c38ca92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b70b2b-fe40-4196-a127-fcbf0c38ca92-plugins-config-volume\") pod \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\" (UID: \"16b70b2b-fe40-4196-a127-fcbf0c38ca92\") "
Oct 31 08:00:08.039009 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.038981    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b70b2b-fe40-4196-a127-fcbf0c38ca92-plugins-config-volume" pod "16b70b2b-fe40-4196-a127-fcbf0c38ca92" (UID: "16b70b2b-fe40-4196-a127-fcbf0c38ca92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:00:08.040580 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.040557    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b70b2b-fe40-4196-a127-fcbf0c38ca92-kube-api-access-jqzvt" pod "16b70b2b-fe40-4196-a127-fcbf0c38ca92" (UID: "16b70b2b-fe40-4196-a127-fcbf0c38ca92"). InnerVolumeSpecName "kube-api-access-jqzvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:00:08.139991 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.139953    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd2db893-738f-4f0a-9313-7227070e6a70-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2db893-738f-4f0a-9313-7227070e6a70-plugins-config-volume\") pod \"cd2db893-738f-4f0a-9313-7227070e6a70\" (UID: \"cd2db893-738f-4f0a-9313-7227070e6a70\") "
Oct 31 08:00:08.140156 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.140031    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd2db893-738f-4f0a-9313-7227070e6a70-kube-api-access-b2xwm\" (UniqueName: \"kubernetes.io/projected/cd2db893-738f-4f0a-9313-7227070e6a70-kube-api-access-b2xwm\") pod \"cd2db893-738f-4f0a-9313-7227070e6a70\" (UID: \"cd2db893-738f-4f0a-9313-7227070e6a70\") "
Oct 31 08:00:08.140266 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.140252    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqzvt\" (UniqueName: \"kubernetes.io/projected/16b70b2b-fe40-4196-a127-fcbf0c38ca92-kube-api-access-jqzvt\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:00:08.140333 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.140268    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16b70b2b-fe40-4196-a127-fcbf0c38ca92-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:00:08.140479 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.140452    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd2db893-738f-4f0a-9313-7227070e6a70-plugins-config-volume" pod "cd2db893-738f-4f0a-9313-7227070e6a70" (UID: "cd2db893-738f-4f0a-9313-7227070e6a70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:00:08.142172 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.142147    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd2db893-738f-4f0a-9313-7227070e6a70-kube-api-access-b2xwm" pod "cd2db893-738f-4f0a-9313-7227070e6a70" (UID: "cd2db893-738f-4f0a-9313-7227070e6a70"). InnerVolumeSpecName "kube-api-access-b2xwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:00:08.241173 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.241085    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xwm\" (UniqueName: \"kubernetes.io/projected/cd2db893-738f-4f0a-9313-7227070e6a70-kube-api-access-b2xwm\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:00:08.241173 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.241116    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd2db893-738f-4f0a-9313-7227070e6a70-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:00:08.744076 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.744042    5829 generic.go:356] "Generic (PLEG): container finished" podID="16b70b2b-fe40-4196-a127-fcbf0c38ca92" containerID="a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58" exitCode=0
Oct 31 08:00:08.744303 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.744101    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"
Oct 31 08:00:08.744303 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.744130    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v" event={"ID":"16b70b2b-fe40-4196-a127-fcbf0c38ca92","Type":"ContainerDied","Data":"a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58"}
Oct 31 08:00:08.744303 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.744188    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v" event={"ID":"16b70b2b-fe40-4196-a127-fcbf0c38ca92","Type":"ContainerDied","Data":"5d4998d604cc55518be0fa3ecc52f4233c25e5ab093346acca59d53e5c37a33b"}
Oct 31 08:00:08.744303 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.744216    5829 scope.go:122] "RemoveContainer" containerID="a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58"
Oct 31 08:00:08.745642 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.745620    5829 generic.go:356] "Generic (PLEG): container finished" podID="cd2db893-738f-4f0a-9313-7227070e6a70" containerID="e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9" exitCode=0
Oct 31 08:00:08.745782 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.745661    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd" event={"ID":"cd2db893-738f-4f0a-9313-7227070e6a70","Type":"ContainerDied","Data":"e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9"}
Oct 31 08:00:08.745782 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.745671    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"
Oct 31 08:00:08.745782 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.745695    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd" event={"ID":"cd2db893-738f-4f0a-9313-7227070e6a70","Type":"ContainerDied","Data":"2f00268e0a6be8fdefc9c4aab039935b5ae04769c8fdb720cfc172c81d78654e"}
Oct 31 08:00:08.755713 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.755693    5829 scope.go:122] "RemoveContainer" containerID="a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58"
Oct 31 08:00:08.756027 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:00:08.756003    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58\": container with ID starting with a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58 not found: ID does not exist" containerID="a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58"
Oct 31 08:00:08.756124 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.756036    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58"} err="failed to get container status \"a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58\": rpc error: code = NotFound desc = could not find container \"a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58\": container with ID starting with a200e43c2d718e54e5207bf4380d73e6f72ac74a592c044f9f4093806218fe58 not found: ID does not exist"
Oct 31 08:00:08.756124 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.756061    5829 scope.go:122] "RemoveContainer" containerID="e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9"
Oct 31 08:00:08.764269 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.764250    5829 scope.go:122] "RemoveContainer" containerID="e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9"
Oct 31 08:00:08.764542 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:00:08.764524    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9\": container with ID starting with e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9 not found: ID does not exist" containerID="e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9"
Oct 31 08:00:08.764591 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.764550    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9"} err="failed to get container status \"e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9\": rpc error: code = NotFound desc = could not find container \"e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9\": container with ID starting with e65839a327669d219921ba997ab43590b16711ba831fad3d180d2268c94b7ab9 not found: ID does not exist"
Oct 31 08:00:08.795982 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.795933    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"]
Oct 31 08:00:08.808711 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.808682    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-11a81e-8546457d9xxx2v"]
Oct 31 08:00:08.843424 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.843391    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"]
Oct 31 08:00:08.848530 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:08.848503    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-11a81e-78d94bx6wd"]
Oct 31 08:00:09.972798 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:09.972762    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b70b2b-fe40-4196-a127-fcbf0c38ca92" path="/var/lib/kubelet/pods/16b70b2b-fe40-4196-a127-fcbf0c38ca92/volumes"
Oct 31 08:00:09.973369 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:09.973349    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd2db893-738f-4f0a-9313-7227070e6a70" path="/var/lib/kubelet/pods/cd2db893-738f-4f0a-9313-7227070e6a70/volumes"
Oct 31 08:00:15.970741 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:15.970708    5829 scope.go:122] "RemoveContainer" containerID="afc122284581539f5de193936dddef2f425d67aafe5071313113a62fa93a1e59"
Oct 31 08:00:16.511160 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:00:16.511121    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:00:16.799330 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:16.799291    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" event={"ID":"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d","Type":"ContainerStarted","Data":"69550b53a38b2a337cd8fb589db9d1e4bf87b2741b599d6a5d1f0f30eeff6932"}
Oct 31 08:00:16.838856 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:16.838807    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" podStartSLOduration=16.838793536 podStartE2EDuration="16.838793536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:00:16.837757513 +0000 UTC m=+1880.976918101" watchObservedRunningTime="2026-10-31 08:00:16.838793536 +0000 UTC m=+1880.977954145"
Oct 31 08:00:17.807060 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:17.807024    5829 generic.go:356] "Generic (PLEG): container finished" podID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerID="69550b53a38b2a337cd8fb589db9d1e4bf87b2741b599d6a5d1f0f30eeff6932" exitCode=60
Oct 31 08:00:17.807534 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:17.807098    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" event={"ID":"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d","Type":"ContainerDied","Data":"69550b53a38b2a337cd8fb589db9d1e4bf87b2741b599d6a5d1f0f30eeff6932"}
Oct 31 08:00:17.807534 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:17.807161    5829 scope.go:122] "RemoveContainer" containerID="afc122284581539f5de193936dddef2f425d67aafe5071313113a62fa93a1e59"
Oct 31 08:00:17.807534 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:17.807512    5829 scope.go:122] "RemoveContainer" containerID="69550b53a38b2a337cd8fb589db9d1e4bf87b2741b599d6a5d1f0f30eeff6932"
Oct 31 08:00:17.807787 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:00:17.807766    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890560-vjqrf_odh-ai-gateway-infra(e7544fbc-ef1f-458b-a5e9-e588ca8cf34d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d"
Oct 31 08:00:17.846583 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:17.846556    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"]
Oct 31 08:00:18.959671 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:18.959643    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:19.047956 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.047921    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-kube-api-access-dczfn\" (UniqueName: \"kubernetes.io/projected/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-kube-api-access-dczfn\") pod \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\" (UID: \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\") "
Oct 31 08:00:19.048127 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.047992    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-maas-api-service-ca\") pod \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\" (UID: \"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d\") "
Oct 31 08:00:19.048491 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.048461    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-maas-api-service-ca" pod "e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" (UID: "e7544fbc-ef1f-458b-a5e9-e588ca8cf34d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:00:19.050020 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.049996    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-kube-api-access-dczfn" pod "e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" (UID: "e7544fbc-ef1f-458b-a5e9-e588ca8cf34d"). InnerVolumeSpecName "kube-api-access-dczfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:00:19.148881 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.148843    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-maas-api-service-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:00:19.148881 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.148884    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dczfn\" (UniqueName: \"kubernetes.io/projected/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d-kube-api-access-dczfn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:00:19.820942 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.820910    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"
Oct 31 08:00:19.821141 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.820900    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf" event={"ID":"e7544fbc-ef1f-458b-a5e9-e588ca8cf34d","Type":"ContainerDied","Data":"36beea4cb4299d29af14430e65ced8bbc6cb6b133f10f1c32f1702129e3bf441"}
Oct 31 08:00:19.821141 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.821066    5829 scope.go:122] "RemoveContainer" containerID="69550b53a38b2a337cd8fb589db9d1e4bf87b2741b599d6a5d1f0f30eeff6932"
Oct 31 08:00:19.872872 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.872834    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"]
Oct 31 08:00:19.885795 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.885760    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890560-vjqrf"]
Oct 31 08:00:19.973360 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:19.973326    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" path="/var/lib/kubelet/pods/e7544fbc-ef1f-458b-a5e9-e588ca8cf34d/volumes"
Oct 31 08:00:24.349482 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:24.349445    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:00:39.877196 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:39.877091    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:00:43.073457 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:43.055339    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"]
Oct 31 08:00:43.073457 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:43.071892    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-sftn4"]
Oct 31 08:00:43.973882 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:43.973846    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c59a7a-9762-4473-953c-bb8ae815eac4" path="/var/lib/kubelet/pods/d3c59a7a-9762-4473-953c-bb8ae815eac4/volumes"
Oct 31 08:00:46.510887 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:00:46.510844    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:00:55.861819 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:55.861782    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:00:58.739465 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:00:58.739414    5829 scope.go:122] "RemoveContainer" containerID="674e4b9c352380df98f8ba61f4b473e3e6b4f61081acbe95f4824781295e310b"
Oct 31 08:01:04.775723 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.775687    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"]
Oct 31 08:01:04.776362 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776342    5829 cpu_manager.go:414] "containerMap: removing container" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerName="cleanup"
Oct 31 08:01:04.776469 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776363    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerName="cleanup"
Oct 31 08:01:04.776469 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776374    5829 cpu_manager.go:414] "containerMap: removing container" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerName="cleanup"
Oct 31 08:01:04.776469 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776382    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerName="cleanup"
Oct 31 08:01:04.776469 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776399    5829 cpu_manager.go:414] "containerMap: removing container" podUID="16b70b2b-fe40-4196-a127-fcbf0c38ca92" containerName="payload-processing"
Oct 31 08:01:04.776469 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776407    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b70b2b-fe40-4196-a127-fcbf0c38ca92" containerName="payload-processing"
Oct 31 08:01:04.776469 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776433    5829 cpu_manager.go:414] "containerMap: removing container" podUID="cd2db893-738f-4f0a-9313-7227070e6a70" containerName="payload-pre-processing"
Oct 31 08:01:04.776469 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776440    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd2db893-738f-4f0a-9313-7227070e6a70" containerName="payload-pre-processing"
Oct 31 08:01:04.776810 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776582    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerName="cleanup"
Oct 31 08:01:04.776810 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776594    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerName="cleanup"
Oct 31 08:01:04.776810 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776607    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b70b2b-fe40-4196-a127-fcbf0c38ca92" containerName="payload-processing"
Oct 31 08:01:04.776810 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.776620    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd2db893-738f-4f0a-9313-7227070e6a70" containerName="payload-pre-processing"
Oct 31 08:01:04.781404 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.781383    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:04.797440 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.797413    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"]
Oct 31 08:01:04.808401 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.808379    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:01:04.808401 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.808378    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 08:01:04.808649 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.808428    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:01:04.808649 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.808518    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:01:04.835843 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.835812    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh4k2\" (UniqueName: \"kubernetes.io/projected/0f97129e-a0b0-493c-b7b0-5b840deb1e66-kube-api-access-zh4k2\") pod \"maas-api-revoke-sub-452b6d7b-k46ml\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:04.836018 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.835924    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-k46ml\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:04.836018 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.835997    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-k46ml\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:04.936704 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.936661    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh4k2\" (UniqueName: \"kubernetes.io/projected/0f97129e-a0b0-493c-b7b0-5b840deb1e66-kube-api-access-zh4k2\") pod \"maas-api-revoke-sub-452b6d7b-k46ml\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:04.936922 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.936802    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-k46ml\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:04.936922 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.936871    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-k46ml\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:04.937545 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.937524    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-k46ml\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:04.939323 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.939299    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-k46ml\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:04.956800 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:04.956769    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh4k2\" (UniqueName: \"kubernetes.io/projected/0f97129e-a0b0-493c-b7b0-5b840deb1e66-kube-api-access-zh4k2\") pod \"maas-api-revoke-sub-452b6d7b-k46ml\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:05.094334 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:05.094220    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:05.390986 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:05.390959    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"]
Oct 31 08:01:05.393609 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:01:05.393580    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f97129e_a0b0_493c_b7b0_5b840deb1e66.slice/crio-c2e5471669a48cd2eaa914e6da596cad684e030d303ee28c6e095af57fff7696 WatchSource:0}: Error finding container c2e5471669a48cd2eaa914e6da596cad684e030d303ee28c6e095af57fff7696: Status 404 returned error can't find the container with id c2e5471669a48cd2eaa914e6da596cad684e030d303ee28c6e095af57fff7696
Oct 31 08:01:06.128805 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:06.128716    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml" event={"ID":"0f97129e-a0b0-493c-b7b0-5b840deb1e66","Type":"ContainerStarted","Data":"d095be5b34a65b26d28e6ecf3a0b00f883431fc31191a5995482c926713d2179"}
Oct 31 08:01:06.128805 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:06.128770    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml" event={"ID":"0f97129e-a0b0-493c-b7b0-5b840deb1e66","Type":"ContainerStarted","Data":"c2e5471669a48cd2eaa914e6da596cad684e030d303ee28c6e095af57fff7696"}
Oct 31 08:01:06.168557 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:06.168502    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml" podStartSLOduration=2.168488336 podStartE2EDuration="2.168488336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:01:06.1673537 +0000 UTC m=+1930.306514293" watchObservedRunningTime="2026-10-31 08:01:06.168488336 +0000 UTC m=+1930.307648923"
Oct 31 08:01:07.135352 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:07.135316    5829 generic.go:356] "Generic (PLEG): container finished" podID="0f97129e-a0b0-493c-b7b0-5b840deb1e66" containerID="d095be5b34a65b26d28e6ecf3a0b00f883431fc31191a5995482c926713d2179" exitCode=0
Oct 31 08:01:07.135753 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:07.135420    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml" event={"ID":"0f97129e-a0b0-493c-b7b0-5b840deb1e66","Type":"ContainerDied","Data":"d095be5b34a65b26d28e6ecf3a0b00f883431fc31191a5995482c926713d2179"}
Oct 31 08:01:08.277220 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.277188    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:08.365173 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.365138    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-xks-ca\") pod \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") "
Oct 31 08:01:08.365388 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.365199    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-openshift-ca\") pod \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") "
Oct 31 08:01:08.365388 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.365345    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f97129e-a0b0-493c-b7b0-5b840deb1e66-kube-api-access-zh4k2\" (UniqueName: \"kubernetes.io/projected/0f97129e-a0b0-493c-b7b0-5b840deb1e66-kube-api-access-zh4k2\") pod \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\" (UID: \"0f97129e-a0b0-493c-b7b0-5b840deb1e66\") "
Oct 31 08:01:08.365610 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.365582    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-openshift-ca" pod "0f97129e-a0b0-493c-b7b0-5b840deb1e66" (UID: "0f97129e-a0b0-493c-b7b0-5b840deb1e66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:01:08.374107 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.374052    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f97129e-a0b0-493c-b7b0-5b840deb1e66-kube-api-access-zh4k2" pod "0f97129e-a0b0-493c-b7b0-5b840deb1e66" (UID: "0f97129e-a0b0-493c-b7b0-5b840deb1e66"). InnerVolumeSpecName "kube-api-access-zh4k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:01:08.374262 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.374052    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-xks-ca" pod "0f97129e-a0b0-493c-b7b0-5b840deb1e66" (UID: "0f97129e-a0b0-493c-b7b0-5b840deb1e66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:01:08.466965 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.466873    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh4k2\" (UniqueName: \"kubernetes.io/projected/0f97129e-a0b0-493c-b7b0-5b840deb1e66-kube-api-access-zh4k2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:01:08.466965 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.466910    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:01:08.466965 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:08.466920    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f97129e-a0b0-493c-b7b0-5b840deb1e66-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:01:09.147087 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:09.147056    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"
Oct 31 08:01:09.147269 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:09.147093    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml" event={"ID":"0f97129e-a0b0-493c-b7b0-5b840deb1e66","Type":"ContainerDied","Data":"c2e5471669a48cd2eaa914e6da596cad684e030d303ee28c6e095af57fff7696"}
Oct 31 08:01:09.147269 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:09.147148    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2e5471669a48cd2eaa914e6da596cad684e030d303ee28c6e095af57fff7696"
Oct 31 08:01:16.511599 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:01:16.511562    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:01:17.753391 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:17.753350    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:01:24.067165 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:24.067127    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:01:30.371822 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:30.371783    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:01:33.779716 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.779674    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"]
Oct 31 08:01:33.780356 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.780335    5829 cpu_manager.go:414] "containerMap: removing container" podUID="0f97129e-a0b0-493c-b7b0-5b840deb1e66" containerName="revoke-keys"
Oct 31 08:01:33.780454 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.780357    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f97129e-a0b0-493c-b7b0-5b840deb1e66" containerName="revoke-keys"
Oct 31 08:01:33.780454 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.780387    5829 cpu_manager.go:414] "containerMap: removing container" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerName="cleanup"
Oct 31 08:01:33.780454 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.780396    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerName="cleanup"
Oct 31 08:01:33.780607 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.780503    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7544fbc-ef1f-458b-a5e9-e588ca8cf34d" containerName="cleanup"
Oct 31 08:01:33.780607 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.780525    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f97129e-a0b0-493c-b7b0-5b840deb1e66" containerName="revoke-keys"
Oct 31 08:01:33.783930 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.783907    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:33.797846 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.797810    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-hpqcq\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:33.797846 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.797856    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-hpqcq\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:33.798146 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.797892    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclcg\" (UniqueName: \"kubernetes.io/projected/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-kube-api-access-hclcg\") pod \"maas-api-revoke-sub-3e31a506-hpqcq\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:33.800680 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.800650    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"]
Oct 31 08:01:33.811374 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.811336    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:01:33.811986 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.811959    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:01:33.812139 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.812112    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 08:01:33.812743 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.812717    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:01:33.898955 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.898910    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-hpqcq\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:33.898955 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.898960    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-hpqcq\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:33.899176 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.898987    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclcg\" (UniqueName: \"kubernetes.io/projected/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-kube-api-access-hclcg\") pod \"maas-api-revoke-sub-3e31a506-hpqcq\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:33.899657 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.899625    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-hpqcq\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:33.901355 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.901331    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-hpqcq\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:33.918443 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:33.918411    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclcg\" (UniqueName: \"kubernetes.io/projected/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-kube-api-access-hclcg\") pod \"maas-api-revoke-sub-3e31a506-hpqcq\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:34.098215 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:34.098117    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:34.397228 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:34.397203    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"]
Oct 31 08:01:34.402807 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:01:34.402775    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b1508e_8c84_4c8e_9a0c_acc65b3089f1.slice/crio-aacedfadfa12a2d7197bc9d04c66934d6b679ac91450732f051f4a7442f93d48 WatchSource:0}: Error finding container aacedfadfa12a2d7197bc9d04c66934d6b679ac91450732f051f4a7442f93d48: Status 404 returned error can't find the container with id aacedfadfa12a2d7197bc9d04c66934d6b679ac91450732f051f4a7442f93d48
Oct 31 08:01:35.356916 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:35.356884    5829 generic.go:356] "Generic (PLEG): container finished" podID="b0b1508e-8c84-4c8e-9a0c-acc65b3089f1" containerID="cadf2db108dd2600d8d80f562b2c0e35cf7bb7f3f467d0ee9368acc3bca1c51a" exitCode=0
Oct 31 08:01:35.357363 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:35.356971    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq" event={"ID":"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1","Type":"ContainerDied","Data":"cadf2db108dd2600d8d80f562b2c0e35cf7bb7f3f467d0ee9368acc3bca1c51a"}
Oct 31 08:01:35.357363 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:35.357041    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq" event={"ID":"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1","Type":"ContainerStarted","Data":"aacedfadfa12a2d7197bc9d04c66934d6b679ac91450732f051f4a7442f93d48"}
Oct 31 08:01:36.506709 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.506683    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:36.525941 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.525912    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-xks-ca\") pod \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") "
Oct 31 08:01:36.526094 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.526026    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-kube-api-access-hclcg\" (UniqueName: \"kubernetes.io/projected/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-kube-api-access-hclcg\") pod \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") "
Oct 31 08:01:36.526094 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.526074    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-openshift-ca\") pod \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\" (UID: \"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1\") "
Oct 31 08:01:36.526478 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.526453    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-openshift-ca" pod "b0b1508e-8c84-4c8e-9a0c-acc65b3089f1" (UID: "b0b1508e-8c84-4c8e-9a0c-acc65b3089f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:01:36.528002 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.527973    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-xks-ca" pod "b0b1508e-8c84-4c8e-9a0c-acc65b3089f1" (UID: "b0b1508e-8c84-4c8e-9a0c-acc65b3089f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:01:36.528089 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.528026    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-kube-api-access-hclcg" pod "b0b1508e-8c84-4c8e-9a0c-acc65b3089f1" (UID: "b0b1508e-8c84-4c8e-9a0c-acc65b3089f1"). InnerVolumeSpecName "kube-api-access-hclcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:01:36.627795 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.627709    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hclcg\" (UniqueName: \"kubernetes.io/projected/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-kube-api-access-hclcg\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:01:36.627795 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.627739    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:01:36.627795 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:36.627751    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:01:37.369120 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:37.369091    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"
Oct 31 08:01:37.369315 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:37.369085    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq" event={"ID":"b0b1508e-8c84-4c8e-9a0c-acc65b3089f1","Type":"ContainerDied","Data":"aacedfadfa12a2d7197bc9d04c66934d6b679ac91450732f051f4a7442f93d48"}
Oct 31 08:01:37.369315 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:37.369221    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aacedfadfa12a2d7197bc9d04c66934d6b679ac91450732f051f4a7442f93d48"
Oct 31 08:01:44.940264 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:44.940226    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:01:46.511598 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:01:46.511557    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:01:58.372458 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:58.372424    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:01:58.837114 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:58.837068    5829 scope.go:122] "RemoveContainer" containerID="1a526e5b697ddba34dd02cd4ebd12aa8e6394456bb1c36d5ced00bfebe0d3f31"
Oct 31 08:01:58.848885 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:01:58.848857    5829 scope.go:122] "RemoveContainer" containerID="f13a7b8b5404f351b19caa1165b839f3f8de01a3cc2a35db16e41d1e4b11c856"
Oct 31 08:02:04.086112 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.086076    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"]
Oct 31 08:02:04.086823 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.086801    5829 cpu_manager.go:414] "containerMap: removing container" podUID="b0b1508e-8c84-4c8e-9a0c-acc65b3089f1" containerName="revoke-keys"
Oct 31 08:02:04.086910 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.086825    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b1508e-8c84-4c8e-9a0c-acc65b3089f1" containerName="revoke-keys"
Oct 31 08:02:04.086968 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.086952    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b1508e-8c84-4c8e-9a0c-acc65b3089f1" containerName="revoke-keys"
Oct 31 08:02:04.091948 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.091924    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.108161 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.108125    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"]
Oct 31 08:02:04.118449 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.118409    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:02:04.118449 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.118409    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 08:02:04.118770 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.118423    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:02:04.120806 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.120783    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:02:04.128443 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.128417    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157e87f5-2jtd9\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.128546 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.128471    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58jx\" (UniqueName: \"kubernetes.io/projected/6cd9045a-c75c-472c-97cf-e5d73b104470-kube-api-access-d58jx\") pod \"maas-api-revoke-sub-157e87f5-2jtd9\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.128587 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.128571    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157e87f5-2jtd9\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.229540 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.229503    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157e87f5-2jtd9\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.229745 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.229622    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157e87f5-2jtd9\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.229745 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.229650    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58jx\" (UniqueName: \"kubernetes.io/projected/6cd9045a-c75c-472c-97cf-e5d73b104470-kube-api-access-d58jx\") pod \"maas-api-revoke-sub-157e87f5-2jtd9\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.230361 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.230322    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157e87f5-2jtd9\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.232026 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.232001    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157e87f5-2jtd9\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.249244 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.249212    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58jx\" (UniqueName: \"kubernetes.io/projected/6cd9045a-c75c-472c-97cf-e5d73b104470-kube-api-access-d58jx\") pod \"maas-api-revoke-sub-157e87f5-2jtd9\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.406570 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.406537    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:04.704513 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:04.704486    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"]
Oct 31 08:02:04.706810 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:02:04.706782    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd9045a_c75c_472c_97cf_e5d73b104470.slice/crio-0f875303b04166a857fca88c25be588c2816f2dc066ac65f73cb817093752660 WatchSource:0}: Error finding container 0f875303b04166a857fca88c25be588c2816f2dc066ac65f73cb817093752660: Status 404 returned error can't find the container with id 0f875303b04166a857fca88c25be588c2816f2dc066ac65f73cb817093752660
Oct 31 08:02:05.534439 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:05.534401    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9" event={"ID":"6cd9045a-c75c-472c-97cf-e5d73b104470","Type":"ContainerStarted","Data":"77c375297c1b2a14bb7f04554b101606ffb640abdb80574e84a8c41c6e76e264"}
Oct 31 08:02:05.534888 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:05.534480    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9" event={"ID":"6cd9045a-c75c-472c-97cf-e5d73b104470","Type":"ContainerStarted","Data":"0f875303b04166a857fca88c25be588c2816f2dc066ac65f73cb817093752660"}
Oct 31 08:02:05.574791 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:05.574737    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9" podStartSLOduration=1.57472215 podStartE2EDuration="1.57472215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:02:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:02:05.572605293 +0000 UTC m=+1989.711765860" watchObservedRunningTime="2026-10-31 08:02:05.57472215 +0000 UTC m=+1989.713882738"
Oct 31 08:02:06.540787 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:06.540740    5829 generic.go:356] "Generic (PLEG): container finished" podID="6cd9045a-c75c-472c-97cf-e5d73b104470" containerID="77c375297c1b2a14bb7f04554b101606ffb640abdb80574e84a8c41c6e76e264" exitCode=0
Oct 31 08:02:06.541197 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:06.540914    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9" event={"ID":"6cd9045a-c75c-472c-97cf-e5d73b104470","Type":"ContainerDied","Data":"77c375297c1b2a14bb7f04554b101606ffb640abdb80574e84a8c41c6e76e264"}
Oct 31 08:02:07.690310 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.690268    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:07.766550 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.766460    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-openshift-ca\") pod \"6cd9045a-c75c-472c-97cf-e5d73b104470\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") "
Oct 31 08:02:07.766550 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.766517    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-xks-ca\") pod \"6cd9045a-c75c-472c-97cf-e5d73b104470\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") "
Oct 31 08:02:07.766807 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.766706    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd9045a-c75c-472c-97cf-e5d73b104470-kube-api-access-d58jx\" (UniqueName: \"kubernetes.io/projected/6cd9045a-c75c-472c-97cf-e5d73b104470-kube-api-access-d58jx\") pod \"6cd9045a-c75c-472c-97cf-e5d73b104470\" (UID: \"6cd9045a-c75c-472c-97cf-e5d73b104470\") "
Oct 31 08:02:07.766893 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.766868    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-openshift-ca" pod "6cd9045a-c75c-472c-97cf-e5d73b104470" (UID: "6cd9045a-c75c-472c-97cf-e5d73b104470"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:02:07.767073 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.767053    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:02:07.768722 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.768691    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-xks-ca" pod "6cd9045a-c75c-472c-97cf-e5d73b104470" (UID: "6cd9045a-c75c-472c-97cf-e5d73b104470"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:02:07.768841 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.768819    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd9045a-c75c-472c-97cf-e5d73b104470-kube-api-access-d58jx" pod "6cd9045a-c75c-472c-97cf-e5d73b104470" (UID: "6cd9045a-c75c-472c-97cf-e5d73b104470"). InnerVolumeSpecName "kube-api-access-d58jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:02:07.867946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.867857    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd9045a-c75c-472c-97cf-e5d73b104470-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:02:07.867946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:07.867891    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58jx\" (UniqueName: \"kubernetes.io/projected/6cd9045a-c75c-472c-97cf-e5d73b104470-kube-api-access-d58jx\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:02:08.552851 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:08.552816    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"
Oct 31 08:02:08.553008 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:08.552849    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9" event={"ID":"6cd9045a-c75c-472c-97cf-e5d73b104470","Type":"ContainerDied","Data":"0f875303b04166a857fca88c25be588c2816f2dc066ac65f73cb817093752660"}
Oct 31 08:02:08.553008 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:08.552896    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f875303b04166a857fca88c25be588c2816f2dc066ac65f73cb817093752660"
Oct 31 08:02:15.368139 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:15.368097    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:02:16.510124 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:02:16.510084    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:02:34.026486 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:34.026441    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"]
Oct 31 08:02:34.038986 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:34.038951    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-9xzhh"]
Oct 31 08:02:35.972723 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:35.972691    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb76d699-99f7-438d-b064-064d7dcb5f2c" path="/var/lib/kubelet/pods/bb76d699-99f7-438d-b064-064d7dcb5f2c/volumes"
Oct 31 08:02:43.624638 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.624597    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"]
Oct 31 08:02:43.625312 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.625259    5829 cpu_manager.go:414] "containerMap: removing container" podUID="6cd9045a-c75c-472c-97cf-e5d73b104470" containerName="revoke-keys"
Oct 31 08:02:43.625425 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.625314    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd9045a-c75c-472c-97cf-e5d73b104470" containerName="revoke-keys"
Oct 31 08:02:43.625483 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.625456    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd9045a-c75c-472c-97cf-e5d73b104470" containerName="revoke-keys"
Oct 31 08:02:43.629046 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.629020    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.643875 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.641368    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb9kn\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-kube-api-access-xb9kn\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.643875 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.641479    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-socket\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.643875 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.641675    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-istio-token\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.643875 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.641727    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-certs\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.643875 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.641762    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-envoy\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.643875 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.641899    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41bdc09-cc97-412e-accb-9a2856137949-istio-podinfo\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.643875 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.641929    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-credential-socket\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.643875 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.641994    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-data\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.643875 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.642034    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41bdc09-cc97-412e-accb-9a2856137949-istiod-ca-cert\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.660908 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.660873    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fd18eeb3-openshift-default-dockercfg-trq5b\""
Oct 31 08:02:43.662919 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.662895    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"]
Oct 31 08:02:43.743489 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.743449    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41bdc09-cc97-412e-accb-9a2856137949-istio-podinfo\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.743754 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.743501    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-credential-socket\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.743754 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.743532    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-data\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.743754 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.743568    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41bdc09-cc97-412e-accb-9a2856137949-istiod-ca-cert\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.743754 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.743623    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb9kn\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-kube-api-access-xb9kn\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.743754 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.743701    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-socket\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.743754 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.743754    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-istio-token\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.744159 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.743793    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-certs\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.744159 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.743820    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-envoy\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.744781 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.744742    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-credential-socket\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.744781 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.744758    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-data\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.745070 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.745047    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-certs\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.745224 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.745060    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-socket\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.745429 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.745408    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41bdc09-cc97-412e-accb-9a2856137949-istiod-ca-cert\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.746680 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.746647    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-envoy\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.752860 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.752829    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41bdc09-cc97-412e-accb-9a2856137949-istio-podinfo\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.764631 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.764594    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-istio-token\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.764796 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.764709    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb9kn\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-kube-api-access-xb9kn\") pod \"e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") " pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:43.954463 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:43.954421    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:44.271697 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:44.271652    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"]
Oct 31 08:02:44.274292 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:02:44.274250    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41bdc09_cc97_412e_accb_9a2856137949.slice/crio-c7664bcb21aa652dccc4d139bdef3c12b7bac08a3099fb8663b4bf42cafd5c11 WatchSource:0}: Error finding container c7664bcb21aa652dccc4d139bdef3c12b7bac08a3099fb8663b4bf42cafd5c11: Status 404 returned error can't find the container with id c7664bcb21aa652dccc4d139bdef3c12b7bac08a3099fb8663b4bf42cafd5c11
Oct 31 08:02:44.276550 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:44.276507    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:02:44.276647 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:44.276579    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:02:44.276647 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:44.276610    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:02:44.759671 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:44.759626    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" event={"ID":"e41bdc09-cc97-412e-accb-9a2856137949","Type":"ContainerStarted","Data":"5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3"}
Oct 31 08:02:44.760051 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:44.759690    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" event={"ID":"e41bdc09-cc97-412e-accb-9a2856137949","Type":"ContainerStarted","Data":"c7664bcb21aa652dccc4d139bdef3c12b7bac08a3099fb8663b4bf42cafd5c11"}
Oct 31 08:02:44.800695 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:44.800641    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" podStartSLOduration=1.800620929 podStartE2EDuration="1.800620929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:02:44.800367639 +0000 UTC m=+2028.939528231" watchObservedRunningTime="2026-10-31 08:02:44.800620929 +0000 UTC m=+2028.939781518"
Oct 31 08:02:44.955518 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:44.955476    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:45.960235 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:45.960202    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:46.511934 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:02:46.511897    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:02:46.771410 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:46.771310    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:46.772381 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:46.772358    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:02:49.525973 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.525935    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"]
Oct 31 08:02:49.530421 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.530393    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:02:49.549872 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.549837    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"]
Oct 31 08:02:49.571297 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.571255    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fd18eeb3\""
Oct 31 08:02:49.582847 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.582822    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fd18eeb3-dockercfg-2fdzs\""
Oct 31 08:02:49.610427 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.610387    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwq9d\" (UniqueName: \"kubernetes.io/projected/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-kube-api-access-gwq9d\") pod \"payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8\" (UID: \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\") " pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:02:49.610636 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.610561    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-plugins-config-volume\") pod \"payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8\" (UID: \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\") " pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:02:49.711343 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.711268    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwq9d\" (UniqueName: \"kubernetes.io/projected/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-kube-api-access-gwq9d\") pod \"payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8\" (UID: \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\") " pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:02:49.711530 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.711393    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-plugins-config-volume\") pod \"payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8\" (UID: \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\") " pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:02:49.712063 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.712043    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-plugins-config-volume\") pod \"payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8\" (UID: \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\") " pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:02:49.731424 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.731392    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwq9d\" (UniqueName: \"kubernetes.io/projected/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-kube-api-access-gwq9d\") pod \"payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8\" (UID: \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\") " pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:02:49.844492 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:49.844414    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:02:50.141313 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:50.141259    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"]
Oct 31 08:02:50.144037 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:02:50.144010    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fbf7dc_21c2_4339_ad21_3d2c4483c81b.slice/crio-ee7f86e86d47dfd020df55509781b53eefb5519b75efd8c1c91a8f65c8437c21 WatchSource:0}: Error finding container ee7f86e86d47dfd020df55509781b53eefb5519b75efd8c1c91a8f65c8437c21: Status 404 returned error can't find the container with id ee7f86e86d47dfd020df55509781b53eefb5519b75efd8c1c91a8f65c8437c21
Oct 31 08:02:50.794596 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:50.794554    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8" event={"ID":"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b","Type":"ContainerStarted","Data":"72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83"}
Oct 31 08:02:50.795333 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:50.794613    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8" event={"ID":"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b","Type":"ContainerStarted","Data":"ee7f86e86d47dfd020df55509781b53eefb5519b75efd8c1c91a8f65c8437c21"}
Oct 31 08:02:50.795333 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:50.794671    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:02:50.834493 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:50.834430    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8" podStartSLOduration=1.8344148219999998 podStartE2EDuration="1.834414822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:02:50.832827085 +0000 UTC m=+2034.971987674" watchObservedRunningTime="2026-10-31 08:02:50.834414822 +0000 UTC m=+2034.973575436"
Oct 31 08:02:53.787292 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:02:53.787256    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:03:01.804263 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:01.804230    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:03:02.812426 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:02.812391    5829 patch_prober.go:29] interesting pod/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 08:03:02.812863 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:02.812460    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:03:03.885722 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:03.885689    5829 generic.go:356] "Generic (PLEG): container finished" podID="e41bdc09-cc97-412e-accb-9a2856137949" containerID="5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3" exitCode=0
Oct 31 08:03:03.886157 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:03.885757    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" event={"ID":"e41bdc09-cc97-412e-accb-9a2856137949","Type":"ContainerDied","Data":"5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3"}
Oct 31 08:03:03.886207 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:03.886163    5829 scope.go:122] "RemoveContainer" containerID="5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3"
Oct 31 08:03:03.887361 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:03.887332    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:03:03.887457 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:03.887417    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:03:03.887505 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:03.887462    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:03:04.893308 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:04.893247    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" event={"ID":"e41bdc09-cc97-412e-accb-9a2856137949","Type":"ContainerStarted","Data":"9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83"}
Oct 31 08:03:04.954761 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:04.954723    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:03:04.955124 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:04.955096    5829 patch_prober.go:29] interesting pod/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 08:03:04.955243 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:04.955154    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 08:03:05.955082 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:05.955037    5829 patch_prober.go:29] interesting pod/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 08:03:05.955566 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:05.955108    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 08:03:06.955211 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:06.955170    5829 patch_prober.go:29] interesting pod/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 08:03:06.955728 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:06.955230    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 08:03:07.954809 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:07.954776    5829 patch_prober.go:29] interesting pod/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 08:03:07.954987 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:07.954845    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 08:03:08.955669 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:08.955632    5829 patch_prober.go:29] interesting pod/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body=
Oct 31 08:03:08.956197 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:08.955691    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused"
Oct 31 08:03:09.959006 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:09.958970    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:03:09.959611 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:09.959384    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:03:09.960091 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:09.960073    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:03:16.509791 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:03:16.509750    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:03:17.406579 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.406524    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"]
Oct 31 08:03:17.411307 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.411269    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.427662 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.427631    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"]
Oct 31 08:03:17.437650 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.437619    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 08:03:17.437827 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.437657    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:03:17.437827 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.437677    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:03:17.440203 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.440183    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:03:17.458476 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.458443    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.458701 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.458674    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.458798 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.458784    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd9t6\" (UniqueName: \"kubernetes.io/projected/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-kube-api-access-cd9t6\") pod \"maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.559634 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.559593    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.560164 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.559655    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd9t6\" (UniqueName: \"kubernetes.io/projected/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-kube-api-access-cd9t6\") pod \"maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.560164 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.559698    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.560466 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.560440    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.562171 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.562139    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.579143 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.579115    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd9t6\" (UniqueName: \"kubernetes.io/projected/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-kube-api-access-cd9t6\") pod \"maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:17.727349 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:17.727230    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:18.027666 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:18.027615    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"]
Oct 31 08:03:18.030360 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:03:18.030329    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9558581_f8d3_4ab2_b9a2_7b7a81db010b.slice/crio-41163a7a0544242fe8bba4cf39344a5fca47a5d8c6f72c9ac200e229c18812e3 WatchSource:0}: Error finding container 41163a7a0544242fe8bba4cf39344a5fca47a5d8c6f72c9ac200e229c18812e3: Status 404 returned error can't find the container with id 41163a7a0544242fe8bba4cf39344a5fca47a5d8c6f72c9ac200e229c18812e3
Oct 31 08:03:19.003271 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:19.003234    5829 generic.go:356] "Generic (PLEG): container finished" podID="d9558581-f8d3-4ab2-b9a2-7b7a81db010b" containerID="0e136b5499a218be9a4a890cae9554606ec66bc34aa70ab81da0d48632ab5a77" exitCode=0
Oct 31 08:03:19.003708 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:19.003316    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87" event={"ID":"d9558581-f8d3-4ab2-b9a2-7b7a81db010b","Type":"ContainerDied","Data":"0e136b5499a218be9a4a890cae9554606ec66bc34aa70ab81da0d48632ab5a77"}
Oct 31 08:03:19.003708 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:19.003412    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87" event={"ID":"d9558581-f8d3-4ab2-b9a2-7b7a81db010b","Type":"ContainerStarted","Data":"41163a7a0544242fe8bba4cf39344a5fca47a5d8c6f72c9ac200e229c18812e3"}
Oct 31 08:03:20.161336 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.161308    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:20.187272 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.187237    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-xks-ca\") pod \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") "
Oct 31 08:03:20.187272 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.187298    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-kube-api-access-cd9t6\" (UniqueName: \"kubernetes.io/projected/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-kube-api-access-cd9t6\") pod \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") "
Oct 31 08:03:20.187539 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.187371    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-openshift-ca\") pod \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\" (UID: \"d9558581-f8d3-4ab2-b9a2-7b7a81db010b\") "
Oct 31 08:03:20.187700 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.187674    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-openshift-ca" pod "d9558581-f8d3-4ab2-b9a2-7b7a81db010b" (UID: "d9558581-f8d3-4ab2-b9a2-7b7a81db010b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:03:20.189414 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.189393    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-xks-ca" pod "d9558581-f8d3-4ab2-b9a2-7b7a81db010b" (UID: "d9558581-f8d3-4ab2-b9a2-7b7a81db010b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:03:20.189547 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.189525    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-kube-api-access-cd9t6" pod "d9558581-f8d3-4ab2-b9a2-7b7a81db010b" (UID: "d9558581-f8d3-4ab2-b9a2-7b7a81db010b"). InnerVolumeSpecName "kube-api-access-cd9t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:03:20.288942 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.288854    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd9t6\" (UniqueName: \"kubernetes.io/projected/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-kube-api-access-cd9t6\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:20.288942 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.288888    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:20.288942 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:20.288899    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9558581-f8d3-4ab2-b9a2-7b7a81db010b-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:21.016904 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:21.016875    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"
Oct 31 08:03:21.017083 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:21.016912    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87" event={"ID":"d9558581-f8d3-4ab2-b9a2-7b7a81db010b","Type":"ContainerDied","Data":"41163a7a0544242fe8bba4cf39344a5fca47a5d8c6f72c9ac200e229c18812e3"}
Oct 31 08:03:21.017083 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:21.016958    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41163a7a0544242fe8bba4cf39344a5fca47a5d8c6f72c9ac200e229c18812e3"
Oct 31 08:03:23.026501 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:23.026469    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"]
Oct 31 08:03:23.038816 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:23.038782    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-g7tm9"]
Oct 31 08:03:23.973099 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:23.973063    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53af56f8-9ea8-4b4e-b703-80b39be83126" path="/var/lib/kubelet/pods/53af56f8-9ea8-4b4e-b703-80b39be83126/volumes"
Oct 31 08:03:27.808943 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:27.808908    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"]
Oct 31 08:03:31.209643 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:31.209600    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"]
Oct 31 08:03:31.222320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:31.222265    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fd18eeb3-fdg87"]
Oct 31 08:03:31.975344 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:31.975308    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9558581-f8d3-4ab2-b9a2-7b7a81db010b" path="/var/lib/kubelet/pods/d9558581-f8d3-4ab2-b9a2-7b7a81db010b/volumes"
Oct 31 08:03:32.571327 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:32.571275    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:03:32.809749 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:32.809707    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8" podUID="f1fbf7dc-21c2-4339-ad21-3d2c4483c81b" containerName="payload-processing" containerID="cri-o://72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83" gracePeriod=25
Oct 31 08:03:33.063342 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.063309    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:03:33.084303 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.084197    5829 generic.go:356] "Generic (PLEG): container finished" podID="f1fbf7dc-21c2-4339-ad21-3d2c4483c81b" containerID="72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83" exitCode=0
Oct 31 08:03:33.084303 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.084239    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"
Oct 31 08:03:33.084303 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.084289    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8" event={"ID":"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b","Type":"ContainerDied","Data":"72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83"}
Oct 31 08:03:33.084524 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.084344    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8" event={"ID":"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b","Type":"ContainerDied","Data":"ee7f86e86d47dfd020df55509781b53eefb5519b75efd8c1c91a8f65c8437c21"}
Oct 31 08:03:33.084524 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.084368    5829 scope.go:122] "RemoveContainer" containerID="72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83"
Oct 31 08:03:33.094214 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.094189    5829 scope.go:122] "RemoveContainer" containerID="72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83"
Oct 31 08:03:33.094551 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:03:33.094530    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83\": container with ID starting with 72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83 not found: ID does not exist" containerID="72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83"
Oct 31 08:03:33.094650 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.094559    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83"} err="failed to get container status \"72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83\": rpc error: code = NotFound desc = could not find container \"72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83\": container with ID starting with 72ade68799c2696527a771bb3e884ace8ed03d4829a33492a9cabb8594176e83 not found: ID does not exist"
Oct 31 08:03:33.121818 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.121786    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-plugins-config-volume\") pod \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\" (UID: \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\") "
Oct 31 08:03:33.122016 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.121849    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-kube-api-access-gwq9d\" (UniqueName: \"kubernetes.io/projected/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-kube-api-access-gwq9d\") pod \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\" (UID: \"f1fbf7dc-21c2-4339-ad21-3d2c4483c81b\") "
Oct 31 08:03:33.122263 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.122233    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-plugins-config-volume" pod "f1fbf7dc-21c2-4339-ad21-3d2c4483c81b" (UID: "f1fbf7dc-21c2-4339-ad21-3d2c4483c81b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:03:33.124206 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.124179    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-kube-api-access-gwq9d" pod "f1fbf7dc-21c2-4339-ad21-3d2c4483c81b" (UID: "f1fbf7dc-21c2-4339-ad21-3d2c4483c81b"). InnerVolumeSpecName "kube-api-access-gwq9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:03:33.222909 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.222869    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:33.222909 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.222904    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwq9d\" (UniqueName: \"kubernetes.io/projected/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b-kube-api-access-gwq9d\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:33.429050 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.429012    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"]
Oct 31 08:03:33.437727 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.437689    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fd18eeb3-854cb6d6d6-r67q8"]
Oct 31 08:03:33.973706 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:33.973668    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fbf7dc-21c2-4339-ad21-3d2c4483c81b" path="/var/lib/kubelet/pods/f1fbf7dc-21c2-4339-ad21-3d2c4483c81b/volumes"
Oct 31 08:03:46.509972 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:03:46.509891    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:03:53.689982 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:53.689943    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"]
Oct 31 08:03:53.690578 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:53.690207    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy" containerID="cri-o://9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83" gracePeriod=30
Oct 31 08:03:54.961902 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:54.961861    5829 patch_prober.go:29] interesting pod/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 08:03:54.962336 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:54.961931    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 08:03:58.196620 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.196585    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 08:03:58.200099 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.200077    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 08:03:58.899964 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.899920    5829 scope.go:122] "RemoveContainer" containerID="ce8e1f13bde12d7855622cc009409bda5f9c8605e1fdf6458bd5bd003ab0345f"
Oct 31 08:03:58.945609 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.945582    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:03:58.973536 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973461    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-socket\") pod \"e41bdc09-cc97-412e-accb-9a2856137949\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") "
Oct 31 08:03:58.973536 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973519    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e41bdc09-cc97-412e-accb-9a2856137949-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41bdc09-cc97-412e-accb-9a2856137949-istiod-ca-cert\") pod \"e41bdc09-cc97-412e-accb-9a2856137949\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") "
Oct 31 08:03:58.973752 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973553    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-envoy\") pod \"e41bdc09-cc97-412e-accb-9a2856137949\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") "
Oct 31 08:03:58.973752 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973584    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e41bdc09-cc97-412e-accb-9a2856137949-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41bdc09-cc97-412e-accb-9a2856137949-istio-podinfo\") pod \"e41bdc09-cc97-412e-accb-9a2856137949\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") "
Oct 31 08:03:58.973752 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973603    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-kube-api-access-xb9kn\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-kube-api-access-xb9kn\") pod \"e41bdc09-cc97-412e-accb-9a2856137949\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") "
Oct 31 08:03:58.973752 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973689    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-data\") pod \"e41bdc09-cc97-412e-accb-9a2856137949\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") "
Oct 31 08:03:58.973752 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973729    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-certs\") pod \"e41bdc09-cc97-412e-accb-9a2856137949\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") "
Oct 31 08:03:58.973752 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973732    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-socket" pod "e41bdc09-cc97-412e-accb-9a2856137949" (UID: "e41bdc09-cc97-412e-accb-9a2856137949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:58.974051 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973777    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-credential-socket\") pod \"e41bdc09-cc97-412e-accb-9a2856137949\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") "
Oct 31 08:03:58.974051 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973830    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-istio-token\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-istio-token\") pod \"e41bdc09-cc97-412e-accb-9a2856137949\" (UID: \"e41bdc09-cc97-412e-accb-9a2856137949\") "
Oct 31 08:03:58.974051 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973972    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-data" pod "e41bdc09-cc97-412e-accb-9a2856137949" (UID: "e41bdc09-cc97-412e-accb-9a2856137949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:58.974051 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.973991    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e41bdc09-cc97-412e-accb-9a2856137949-istiod-ca-cert" pod "e41bdc09-cc97-412e-accb-9a2856137949" (UID: "e41bdc09-cc97-412e-accb-9a2856137949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:03:58.974267 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.974192    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:58.974267 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.974212    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:58.974267 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.974207    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-credential-socket" pod "e41bdc09-cc97-412e-accb-9a2856137949" (UID: "e41bdc09-cc97-412e-accb-9a2856137949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:58.974267 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.974219    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-certs" pod "e41bdc09-cc97-412e-accb-9a2856137949" (UID: "e41bdc09-cc97-412e-accb-9a2856137949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:58.974267 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.974227    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e41bdc09-cc97-412e-accb-9a2856137949-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:58.975825 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.975806    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-envoy" pod "e41bdc09-cc97-412e-accb-9a2856137949" (UID: "e41bdc09-cc97-412e-accb-9a2856137949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:03:58.976094 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.976077    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e41bdc09-cc97-412e-accb-9a2856137949-istio-podinfo" pod "e41bdc09-cc97-412e-accb-9a2856137949" (UID: "e41bdc09-cc97-412e-accb-9a2856137949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 08:03:58.976357 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.976329    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-kube-api-access-xb9kn" pod "e41bdc09-cc97-412e-accb-9a2856137949" (UID: "e41bdc09-cc97-412e-accb-9a2856137949"). InnerVolumeSpecName "kube-api-access-xb9kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:03:58.976442 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:58.976390    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-istio-token" pod "e41bdc09-cc97-412e-accb-9a2856137949" (UID: "e41bdc09-cc97-412e-accb-9a2856137949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:03:59.075430 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.075399    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:59.075690 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.075664    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e41bdc09-cc97-412e-accb-9a2856137949-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:59.075804 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.075693    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb9kn\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-kube-api-access-xb9kn\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:59.075804 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.075713    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:59.075804 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.075729    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e41bdc09-cc97-412e-accb-9a2856137949-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:59.075804 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.075746    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e41bdc09-cc97-412e-accb-9a2856137949-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:03:59.228051 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.227961    5829 generic.go:356] "Generic (PLEG): container finished" podID="e41bdc09-cc97-412e-accb-9a2856137949" containerID="9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83" exitCode=0
Oct 31 08:03:59.228051 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.228016    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"
Oct 31 08:03:59.228051 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.228034    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" event={"ID":"e41bdc09-cc97-412e-accb-9a2856137949","Type":"ContainerDied","Data":"9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83"}
Oct 31 08:03:59.228641 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.228089    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr" event={"ID":"e41bdc09-cc97-412e-accb-9a2856137949","Type":"ContainerDied","Data":"c7664bcb21aa652dccc4d139bdef3c12b7bac08a3099fb8663b4bf42cafd5c11"}
Oct 31 08:03:59.228641 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.228113    5829 scope.go:122] "RemoveContainer" containerID="9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83"
Oct 31 08:03:59.237932 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.237914    5829 scope.go:122] "RemoveContainer" containerID="5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3"
Oct 31 08:03:59.246952 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.246931    5829 scope.go:122] "RemoveContainer" containerID="9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83"
Oct 31 08:03:59.247275 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:03:59.247253    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83\": container with ID starting with 9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83 not found: ID does not exist" containerID="9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83"
Oct 31 08:03:59.247348 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.247312    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83"} err="failed to get container status \"9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83\": rpc error: code = NotFound desc = could not find container \"9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83\": container with ID starting with 9f06aeb31e1a788dee436d9feeee444a793ee058e765dd4a4281d49b52d5fa83 not found: ID does not exist"
Oct 31 08:03:59.247348 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.247332    5829 scope.go:122] "RemoveContainer" containerID="5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3"
Oct 31 08:03:59.247601 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:03:59.247582    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3\": container with ID starting with 5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3 not found: ID does not exist" containerID="5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3"
Oct 31 08:03:59.247656 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.247610    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3"} err="failed to get container status \"5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3\": rpc error: code = NotFound desc = could not find container \"5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3\": container with ID starting with 5da3a5263da4c7cabb48f2907b290fe7e07339736668faed13de5ba4aab3a4a3 not found: ID does not exist"
Oct 31 08:03:59.279644 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.279609    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"]
Oct 31 08:03:59.293054 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.293021    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fd18eeb3-openshift-default-65fbf6c9cc-7sbgr"]
Oct 31 08:03:59.973070 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:03:59.973036    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e41bdc09-cc97-412e-accb-9a2856137949" path="/var/lib/kubelet/pods/e41bdc09-cc97-412e-accb-9a2856137949/volumes"
Oct 31 08:04:05.443912 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.443842    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"]
Oct 31 08:04:05.444495 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444470    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d9558581-f8d3-4ab2-b9a2-7b7a81db010b" containerName="revoke-keys"
Oct 31 08:04:05.444495 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444493    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9558581-f8d3-4ab2-b9a2-7b7a81db010b" containerName="revoke-keys"
Oct 31 08:04:05.444679 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444526    5829 cpu_manager.go:414] "containerMap: removing container" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy"
Oct 31 08:04:05.444679 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444532    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy"
Oct 31 08:04:05.444679 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444548    5829 cpu_manager.go:414] "containerMap: removing container" podUID="f1fbf7dc-21c2-4339-ad21-3d2c4483c81b" containerName="payload-processing"
Oct 31 08:04:05.444679 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444554    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fbf7dc-21c2-4339-ad21-3d2c4483c81b" containerName="payload-processing"
Oct 31 08:04:05.444679 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444595    5829 cpu_manager.go:414] "containerMap: removing container" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy"
Oct 31 08:04:05.444679 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444602    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy"
Oct 31 08:04:05.444946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444687    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fbf7dc-21c2-4339-ad21-3d2c4483c81b" containerName="payload-processing"
Oct 31 08:04:05.444946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444699    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy"
Oct 31 08:04:05.444946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444706    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9558581-f8d3-4ab2-b9a2-7b7a81db010b" containerName="revoke-keys"
Oct 31 08:04:05.444946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.444716    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="e41bdc09-cc97-412e-accb-9a2856137949" containerName="istio-proxy"
Oct 31 08:04:05.449885 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.449865    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.479124 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.479092    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-0503df-openshift-default-dockercfg-27zvm\""
Oct 31 08:04:05.481107 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.481078    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"]
Oct 31 08:04:05.542746 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.542694    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690eb53-1c5d-47e3-b41d-39b608260e69-istiod-ca-cert\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.542746 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.542749    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-credential-socket\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.542968 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.542869    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-certs\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.542968 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.542933    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-socket\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.542968 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.542960    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-podinfo\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.543086 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.542998    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-token\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.543149 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.543131    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-envoy\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.543186 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.543168    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkb8r\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-kube-api-access-zkb8r\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.543246 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.543230    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-data\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.644698 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.644656    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-envoy\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.644698 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.644705    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkb8r\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-kube-api-access-zkb8r\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645059 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.644732    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-data\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645059 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.644756    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690eb53-1c5d-47e3-b41d-39b608260e69-istiod-ca-cert\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645059 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.644778    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-credential-socket\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645059 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.644821    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-certs\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645059 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.644869    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-socket\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645059 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.644903    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-podinfo\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645059 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.644955    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-token\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645448 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.645218    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-credential-socket\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645448 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.645258    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-certs\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645561 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.645475    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-socket\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645699 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.645674    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690eb53-1c5d-47e3-b41d-39b608260e69-istiod-ca-cert\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.645773 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.645745    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-data\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.647130 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.647110    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-envoy\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.647568 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.647551    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-podinfo\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.664431 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.664404    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkb8r\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-kube-api-access-zkb8r\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.664431 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.664412    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-token\") pod \"e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") " pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:05.763501 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:05.763412    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:06.078059 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:06.078034    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"]
Oct 31 08:04:06.080488 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:04:06.080463    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc690eb53_1c5d_47e3_b41d_39b608260e69.slice/crio-bd5cdd2899ecbd4e36d9f9468493bc43f87f6b06d3d16ff671f227a0973166b8 WatchSource:0}: Error finding container bd5cdd2899ecbd4e36d9f9468493bc43f87f6b06d3d16ff671f227a0973166b8: Status 404 returned error can't find the container with id bd5cdd2899ecbd4e36d9f9468493bc43f87f6b06d3d16ff671f227a0973166b8
Oct 31 08:04:06.082654 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:06.082623    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:04:06.082723 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:06.082687    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:04:06.082723 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:06.082716    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:04:06.269784 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:06.269740    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" event={"ID":"c690eb53-1c5d-47e3-b41d-39b608260e69","Type":"ContainerStarted","Data":"66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18"}
Oct 31 08:04:06.269960 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:06.269820    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" event={"ID":"c690eb53-1c5d-47e3-b41d-39b608260e69","Type":"ContainerStarted","Data":"bd5cdd2899ecbd4e36d9f9468493bc43f87f6b06d3d16ff671f227a0973166b8"}
Oct 31 08:04:06.312456 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:06.312394    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" podStartSLOduration=1.312376175 podStartE2EDuration="1.312376175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:04:06.310122981 +0000 UTC m=+2110.449283571" watchObservedRunningTime="2026-10-31 08:04:06.312376175 +0000 UTC m=+2110.451536763"
Oct 31 08:04:06.764534 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:06.764494    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:07.769794 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:07.769759    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:08.282458 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:08.282427    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:08.283414 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:08.283395    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:10.408733 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.408699    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"]
Oct 31 08:04:10.412897 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.412875    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:10.433260 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.433227    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"]
Oct 31 08:04:10.441774 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.441744    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-0503df-dockercfg-8nrz7\""
Oct 31 08:04:10.441915 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.441773    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-0503df\""
Oct 31 08:04:10.457157 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.457129    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"]
Oct 31 08:04:10.461032 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.461008    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:10.478441 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.478411    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"]
Oct 31 08:04:10.494573 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.494540    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6362a4d9-817a-418f-a08d-1219d825dc73-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl\" (UID: \"6362a4d9-817a-418f-a08d-1219d825dc73\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:10.494733 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.494593    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzclb\" (UniqueName: \"kubernetes.io/projected/6362a4d9-817a-418f-a08d-1219d825dc73-kube-api-access-lzclb\") pod \"payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl\" (UID: \"6362a4d9-817a-418f-a08d-1219d825dc73\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:10.595564 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.595518    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6445859f-f36d-4fac-93e5-153838394b00-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb\" (UID: \"6445859f-f36d-4fac-93e5-153838394b00\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:10.595762 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.595705    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6362a4d9-817a-418f-a08d-1219d825dc73-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl\" (UID: \"6362a4d9-817a-418f-a08d-1219d825dc73\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:10.595762 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.595749    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzbh\" (UniqueName: \"kubernetes.io/projected/6445859f-f36d-4fac-93e5-153838394b00-kube-api-access-cnzbh\") pod \"payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb\" (UID: \"6445859f-f36d-4fac-93e5-153838394b00\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:10.595884 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.595776    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzclb\" (UniqueName: \"kubernetes.io/projected/6362a4d9-817a-418f-a08d-1219d825dc73-kube-api-access-lzclb\") pod \"payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl\" (UID: \"6362a4d9-817a-418f-a08d-1219d825dc73\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:10.596483 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.596457    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6362a4d9-817a-418f-a08d-1219d825dc73-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl\" (UID: \"6362a4d9-817a-418f-a08d-1219d825dc73\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:10.619452 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.619420    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzclb\" (UniqueName: \"kubernetes.io/projected/6362a4d9-817a-418f-a08d-1219d825dc73-kube-api-access-lzclb\") pod \"payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl\" (UID: \"6362a4d9-817a-418f-a08d-1219d825dc73\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:10.696469 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.696378    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6445859f-f36d-4fac-93e5-153838394b00-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb\" (UID: \"6445859f-f36d-4fac-93e5-153838394b00\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:10.696615 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.696511    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzbh\" (UniqueName: \"kubernetes.io/projected/6445859f-f36d-4fac-93e5-153838394b00-kube-api-access-cnzbh\") pod \"payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb\" (UID: \"6445859f-f36d-4fac-93e5-153838394b00\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:10.697118 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.697099    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6445859f-f36d-4fac-93e5-153838394b00-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb\" (UID: \"6445859f-f36d-4fac-93e5-153838394b00\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:10.716425 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.716396    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzbh\" (UniqueName: \"kubernetes.io/projected/6445859f-f36d-4fac-93e5-153838394b00-kube-api-access-cnzbh\") pod \"payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb\" (UID: \"6445859f-f36d-4fac-93e5-153838394b00\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:10.728240 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.728212    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:10.774988 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:10.774950    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:11.029112 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.029034    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"]
Oct 31 08:04:11.032415 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:04:11.032382    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6362a4d9_817a_418f_a08d_1219d825dc73.slice/crio-2a4bfc1f548fa5f680ee1289375ba334aaa10863999689abce0e07ac35d1d0ab WatchSource:0}: Error finding container 2a4bfc1f548fa5f680ee1289375ba334aaa10863999689abce0e07ac35d1d0ab: Status 404 returned error can't find the container with id 2a4bfc1f548fa5f680ee1289375ba334aaa10863999689abce0e07ac35d1d0ab
Oct 31 08:04:11.088463 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.088438    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"]
Oct 31 08:04:11.090437 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:04:11.090412    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6445859f_f36d_4fac_93e5_153838394b00.slice/crio-9404b4fccdc9b8dd71c8b64ef09a53c21393120af0f23071893a3a7ddf6f5806 WatchSource:0}: Error finding container 9404b4fccdc9b8dd71c8b64ef09a53c21393120af0f23071893a3a7ddf6f5806: Status 404 returned error can't find the container with id 9404b4fccdc9b8dd71c8b64ef09a53c21393120af0f23071893a3a7ddf6f5806
Oct 31 08:04:11.307034 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.306924    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl" event={"ID":"6362a4d9-817a-418f-a08d-1219d825dc73","Type":"ContainerStarted","Data":"d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039"}
Oct 31 08:04:11.307034 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.307019    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl" event={"ID":"6362a4d9-817a-418f-a08d-1219d825dc73","Type":"ContainerStarted","Data":"2a4bfc1f548fa5f680ee1289375ba334aaa10863999689abce0e07ac35d1d0ab"}
Oct 31 08:04:11.307271 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.307049    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:11.308821 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.308784    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb" event={"ID":"6445859f-f36d-4fac-93e5-153838394b00","Type":"ContainerStarted","Data":"29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3"}
Oct 31 08:04:11.308929 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.308844    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb" event={"ID":"6445859f-f36d-4fac-93e5-153838394b00","Type":"ContainerStarted","Data":"9404b4fccdc9b8dd71c8b64ef09a53c21393120af0f23071893a3a7ddf6f5806"}
Oct 31 08:04:11.308929 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.308874    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:11.349772 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.349709    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl" podStartSLOduration=1.349689989 podStartE2EDuration="1.349689989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:04:11.346870514 +0000 UTC m=+2115.486031102" watchObservedRunningTime="2026-10-31 08:04:11.349689989 +0000 UTC m=+2115.488850578"
Oct 31 08:04:11.389075 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:11.389027    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb" podStartSLOduration=1.3890111410000001 podStartE2EDuration="1.389011141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:04:11.387392297 +0000 UTC m=+2115.526552885" watchObservedRunningTime="2026-10-31 08:04:11.389011141 +0000 UTC m=+2115.528171729"
Oct 31 08:04:13.389795 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:13.389754    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:04:16.532758 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:04:16.532718    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:04:22.317977 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:22.317946    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:04:22.318504 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:22.318011    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:04:23.283721 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:23.283686    5829 patch_prober.go:29] interesting pod/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 08:04:23.283950 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:23.283752    5829 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 08:04:23.396739 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:23.396707    5829 generic.go:356] "Generic (PLEG): container finished" podID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerID="66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18" exitCode=0
Oct 31 08:04:23.397134 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:23.396774    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" event={"ID":"c690eb53-1c5d-47e3-b41d-39b608260e69","Type":"ContainerDied","Data":"66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18"}
Oct 31 08:04:23.397189 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:23.397172    5829 scope.go:122] "RemoveContainer" containerID="66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18"
Oct 31 08:04:23.398502 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:23.398475    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:04:23.398560 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:23.398543    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:04:23.398598 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:23.398576    5829 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 31 08:04:24.429372 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:24.429337    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" event={"ID":"c690eb53-1c5d-47e3-b41d-39b608260e69","Type":"ContainerStarted","Data":"8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468"}
Oct 31 08:04:24.763895 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:24.763813    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:24.764065 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:24.763990    5829 patch_prober.go:29] interesting pod/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 08:04:24.764065 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:24.764034    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 08:04:25.028363 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:25.027998    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"]
Oct 31 08:04:25.031467 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:25.031435    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef47abfe-h4hhq"]
Oct 31 08:04:25.763858 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:25.763821    5829 patch_prober.go:29] interesting pod/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 08:04:25.764268 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:25.763881    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 08:04:25.973455 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:25.973410    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21" path="/var/lib/kubelet/pods/fc28bb60-e2f2-4465-8dfb-b7e65b9fdb21/volumes"
Oct 31 08:04:26.764433 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:26.764401    5829 patch_prober.go:29] interesting pod/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 08:04:26.764827 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:26.764454    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 08:04:27.763889 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:27.763853    5829 patch_prober.go:29] interesting pod/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 08:04:27.764085 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:27.763907    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 08:04:28.764223 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:28.764187    5829 patch_prober.go:29] interesting pod/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body=
Oct 31 08:04:28.764664 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:28.764245    5829 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused"
Oct 31 08:04:29.769366 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:29.769336    5829 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:29.769767 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:29.769696    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:29.770516 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:29.770495    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:04:46.511508 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:04:46.511464    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:04:58.933300 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:04:58.933260    5829 scope.go:122] "RemoveContainer" containerID="66ae4b182d51d8c2254d941503a14d9b54afaa9211bba64331bb0ab2ecaa0ead"
Oct 31 08:05:16.511622 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:05:16.511527    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:05:43.975323 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:43.975276    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"]
Oct 31 08:05:43.979059 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:43.979035    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:43.995637 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:43.995609    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"]
Oct 31 08:05:44.006254 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.006220    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 08:05:44.006254 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.006251    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:05:44.006466 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.006357    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:05:44.007167 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.007152    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:05:44.030911 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.030859    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpxhh\" (UniqueName: \"kubernetes.io/projected/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-kube-api-access-jpxhh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.031193 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.031161    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.031320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.031255    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.132397 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.132354    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.132397 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.132403    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.132627 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.132503    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpxhh\" (UniqueName: \"kubernetes.io/projected/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-kube-api-access-jpxhh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.133029 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.133004    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.134868 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.134840    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.152015 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.151982    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpxhh\" (UniqueName: \"kubernetes.io/projected/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-kube-api-access-jpxhh\") pod \"maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.291586 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.291501    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:44.588121 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.588088    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"]
Oct 31 08:05:44.590999 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:05:44.590963    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c624b3_cbe6_4ea3_86b4_e5ac8f6c7d13.slice/crio-0040b5b756d6a25a859fa7f2c360919490d5fae1b320558bcb1c7352acf0ced9 WatchSource:0}: Error finding container 0040b5b756d6a25a859fa7f2c360919490d5fae1b320558bcb1c7352acf0ced9: Status 404 returned error can't find the container with id 0040b5b756d6a25a859fa7f2c360919490d5fae1b320558bcb1c7352acf0ced9
Oct 31 08:05:44.878802 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:44.878715    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt" event={"ID":"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13","Type":"ContainerStarted","Data":"0040b5b756d6a25a859fa7f2c360919490d5fae1b320558bcb1c7352acf0ced9"}
Oct 31 08:05:45.887173 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:45.887140    5829 generic.go:356] "Generic (PLEG): container finished" podID="d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13" containerID="fad1eb3c718b728310ea7d5a10ac7245889afd0375d7efdca7906333bbd73e39" exitCode=0
Oct 31 08:05:45.887592 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:45.887222    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt" event={"ID":"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13","Type":"ContainerDied","Data":"fad1eb3c718b728310ea7d5a10ac7245889afd0375d7efdca7906333bbd73e39"}
Oct 31 08:05:46.511179 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:05:46.511144    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:05:47.025885 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.025860    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:47.064340 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.064276    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-openshift-ca\") pod \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") "
Oct 31 08:05:47.064545 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.064371    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-kube-api-access-jpxhh\" (UniqueName: \"kubernetes.io/projected/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-kube-api-access-jpxhh\") pod \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") "
Oct 31 08:05:47.064545 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.064469    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-xks-ca\") pod \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\" (UID: \"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13\") "
Oct 31 08:05:47.068378 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.066831    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-openshift-ca" pod "d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13" (UID: "d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:05:47.080434 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.080376    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-xks-ca" pod "d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13" (UID: "d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:05:47.080936 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.080898    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-kube-api-access-jpxhh" pod "d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13" (UID: "d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13"). InnerVolumeSpecName "kube-api-access-jpxhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:05:47.168023 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.167922    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:05:47.168023 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.167955    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpxhh\" (UniqueName: \"kubernetes.io/projected/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-kube-api-access-jpxhh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:05:47.168023 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.167966    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:05:47.899937 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.899903    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"
Oct 31 08:05:47.900095 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.899945    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt" event={"ID":"d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13","Type":"ContainerDied","Data":"0040b5b756d6a25a859fa7f2c360919490d5fae1b320558bcb1c7352acf0ced9"}
Oct 31 08:05:47.900095 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:47.900021    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0040b5b756d6a25a859fa7f2c360919490d5fae1b320558bcb1c7352acf0ced9"
Oct 31 08:05:54.369857 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:54.369769    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"]
Oct 31 08:05:54.399700 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:54.399664    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"]
Oct 31 08:05:58.960294 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:58.960247    5829 scope.go:122] "RemoveContainer" containerID="f4f7b904fa413486dc690f9bcd11352fc4fa82258ae6cf87d84764764a02a0e0"
Oct 31 08:05:58.969367 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:58.969341    5829 scope.go:122] "RemoveContainer" containerID="a9892d699418687297dc5b38c0f3ab5dd3d7b9fea70e8b07462f5bac0931e9a2"
Oct 31 08:05:59.370901 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.370804    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl" podUID="6362a4d9-817a-418f-a08d-1219d825dc73" containerName="payload-processing" containerID="cri-o://d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039" gracePeriod=25
Oct 31 08:05:59.400706 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.400643    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb" podUID="6445859f-f36d-4fac-93e5-153838394b00" containerName="payload-pre-processing" containerID="cri-o://29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3" gracePeriod=25
Oct 31 08:05:59.647641 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.647611    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:05:59.686193 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.686160    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:05:59.807669 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.807638    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6362a4d9-817a-418f-a08d-1219d825dc73-kube-api-access-lzclb\" (UniqueName: \"kubernetes.io/projected/6362a4d9-817a-418f-a08d-1219d825dc73-kube-api-access-lzclb\") pod \"6362a4d9-817a-418f-a08d-1219d825dc73\" (UID: \"6362a4d9-817a-418f-a08d-1219d825dc73\") "
Oct 31 08:05:59.807882 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.807696    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6445859f-f36d-4fac-93e5-153838394b00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6445859f-f36d-4fac-93e5-153838394b00-plugins-config-volume\") pod \"6445859f-f36d-4fac-93e5-153838394b00\" (UID: \"6445859f-f36d-4fac-93e5-153838394b00\") "
Oct 31 08:05:59.807882 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.807770    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6362a4d9-817a-418f-a08d-1219d825dc73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6362a4d9-817a-418f-a08d-1219d825dc73-plugins-config-volume\") pod \"6362a4d9-817a-418f-a08d-1219d825dc73\" (UID: \"6362a4d9-817a-418f-a08d-1219d825dc73\") "
Oct 31 08:05:59.807882 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.807878    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6445859f-f36d-4fac-93e5-153838394b00-kube-api-access-cnzbh\" (UniqueName: \"kubernetes.io/projected/6445859f-f36d-4fac-93e5-153838394b00-kube-api-access-cnzbh\") pod \"6445859f-f36d-4fac-93e5-153838394b00\" (UID: \"6445859f-f36d-4fac-93e5-153838394b00\") "
Oct 31 08:05:59.808117 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.808082    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6445859f-f36d-4fac-93e5-153838394b00-plugins-config-volume" pod "6445859f-f36d-4fac-93e5-153838394b00" (UID: "6445859f-f36d-4fac-93e5-153838394b00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:05:59.808264 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.808122    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6362a4d9-817a-418f-a08d-1219d825dc73-plugins-config-volume" pod "6362a4d9-817a-418f-a08d-1219d825dc73" (UID: "6362a4d9-817a-418f-a08d-1219d825dc73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:05:59.808369 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.808306    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6445859f-f36d-4fac-93e5-153838394b00-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:05:59.808369 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.808328    5829 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6362a4d9-817a-418f-a08d-1219d825dc73-plugins-config-volume\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:05:59.809896 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.809867    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6362a4d9-817a-418f-a08d-1219d825dc73-kube-api-access-lzclb" pod "6362a4d9-817a-418f-a08d-1219d825dc73" (UID: "6362a4d9-817a-418f-a08d-1219d825dc73"). InnerVolumeSpecName "kube-api-access-lzclb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:05:59.810106 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.810079    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6445859f-f36d-4fac-93e5-153838394b00-kube-api-access-cnzbh" pod "6445859f-f36d-4fac-93e5-153838394b00" (UID: "6445859f-f36d-4fac-93e5-153838394b00"). InnerVolumeSpecName "kube-api-access-cnzbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:05:59.908947 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.908906    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzbh\" (UniqueName: \"kubernetes.io/projected/6445859f-f36d-4fac-93e5-153838394b00-kube-api-access-cnzbh\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:05:59.908947 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.908944    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzclb\" (UniqueName: \"kubernetes.io/projected/6362a4d9-817a-418f-a08d-1219d825dc73-kube-api-access-lzclb\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:05:59.968560 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.968529    5829 generic.go:356] "Generic (PLEG): container finished" podID="6362a4d9-817a-418f-a08d-1219d825dc73" containerID="d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039" exitCode=0
Oct 31 08:05:59.968998 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.968656    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"
Oct 31 08:05:59.970206 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.970176    5829 generic.go:356] "Generic (PLEG): container finished" podID="6445859f-f36d-4fac-93e5-153838394b00" containerID="29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3" exitCode=0
Oct 31 08:05:59.970366 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.970276    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"
Oct 31 08:05:59.976064 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.976034    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl" event={"ID":"6362a4d9-817a-418f-a08d-1219d825dc73","Type":"ContainerDied","Data":"d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039"}
Oct 31 08:05:59.976187 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.976089    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl" event={"ID":"6362a4d9-817a-418f-a08d-1219d825dc73","Type":"ContainerDied","Data":"2a4bfc1f548fa5f680ee1289375ba334aaa10863999689abce0e07ac35d1d0ab"}
Oct 31 08:05:59.976187 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.976117    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb" event={"ID":"6445859f-f36d-4fac-93e5-153838394b00","Type":"ContainerDied","Data":"29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3"}
Oct 31 08:05:59.976187 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.976141    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb" event={"ID":"6445859f-f36d-4fac-93e5-153838394b00","Type":"ContainerDied","Data":"9404b4fccdc9b8dd71c8b64ef09a53c21393120af0f23071893a3a7ddf6f5806"}
Oct 31 08:05:59.976187 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.976161    5829 scope.go:122] "RemoveContainer" containerID="d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039"
Oct 31 08:05:59.987016 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.987000    5829 scope.go:122] "RemoveContainer" containerID="d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039"
Oct 31 08:05:59.987292 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:05:59.987264    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039\": container with ID starting with d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039 not found: ID does not exist" containerID="d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039"
Oct 31 08:05:59.987337 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.987302    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039"} err="failed to get container status \"d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039\": rpc error: code = NotFound desc = could not find container \"d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039\": container with ID starting with d141dc521f1dce942599e3c8aa275ab07db12499f1b1a083418718001a692039 not found: ID does not exist"
Oct 31 08:05:59.987337 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.987325    5829 scope.go:122] "RemoveContainer" containerID="29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3"
Oct 31 08:05:59.995818 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.995788    5829 scope.go:122] "RemoveContainer" containerID="29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3"
Oct 31 08:05:59.996095 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:05:59.996075    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3\": container with ID starting with 29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3 not found: ID does not exist" containerID="29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3"
Oct 31 08:05:59.996145 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:05:59.996103    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3"} err="failed to get container status \"29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3\": rpc error: code = NotFound desc = could not find container \"29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3\": container with ID starting with 29c6b3391fc5d35fc256c97a7c49f8a4a3f0e68662d585eaaadf508f0355c7f3 not found: ID does not exist"
Oct 31 08:06:00.035671 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:00.035633    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"]
Oct 31 08:06:00.041148 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:00.041073    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-0503df-7879ddd84-rjdzl"]
Oct 31 08:06:00.072882 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:00.072838    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"]
Oct 31 08:06:00.076375 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:00.076343    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-0503df-78657cc5cf-8kcqb"]
Oct 31 08:06:00.499350 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:00.499311    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"]
Oct 31 08:06:00.505433 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:00.505391    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-0503df-5hvgt"]
Oct 31 08:06:01.973753 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:01.973717    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6362a4d9-817a-418f-a08d-1219d825dc73" path="/var/lib/kubelet/pods/6362a4d9-817a-418f-a08d-1219d825dc73/volumes"
Oct 31 08:06:01.974161 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:01.974119    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6445859f-f36d-4fac-93e5-153838394b00" path="/var/lib/kubelet/pods/6445859f-f36d-4fac-93e5-153838394b00/volumes"
Oct 31 08:06:01.974575 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:01.974560    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13" path="/var/lib/kubelet/pods/d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13/volumes"
Oct 31 08:06:02.079444 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:02.079399    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:06:07.411545 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:07.411513    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"]
Oct 31 08:06:07.412084 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:07.411725    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy" containerID="cri-o://8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468" gracePeriod=30
Oct 31 08:06:09.025072 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:09.025039    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"]
Oct 31 08:06:09.037670 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:09.037637    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-k46ml"]
Oct 31 08:06:09.877020 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:09.876985    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:06:09.973359 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:09.973324    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f97129e-a0b0-493c-b7b0-5b840deb1e66" path="/var/lib/kubelet/pods/0f97129e-a0b0-493c-b7b0-5b840deb1e66/volumes"
Oct 31 08:06:12.806190 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.806154    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:06:12.894400 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894355    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-podinfo\") pod \"c690eb53-1c5d-47e3-b41d-39b608260e69\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") "
Oct 31 08:06:12.894604 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894432    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-socket\") pod \"c690eb53-1c5d-47e3-b41d-39b608260e69\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") "
Oct 31 08:06:12.894604 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894460    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-data\") pod \"c690eb53-1c5d-47e3-b41d-39b608260e69\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") "
Oct 31 08:06:12.894604 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894485    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-credential-socket\") pod \"c690eb53-1c5d-47e3-b41d-39b608260e69\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") "
Oct 31 08:06:12.894604 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894535    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c690eb53-1c5d-47e3-b41d-39b608260e69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690eb53-1c5d-47e3-b41d-39b608260e69-istiod-ca-cert\") pod \"c690eb53-1c5d-47e3-b41d-39b608260e69\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") "
Oct 31 08:06:12.894604 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894558    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-kube-api-access-zkb8r\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-kube-api-access-zkb8r\") pod \"c690eb53-1c5d-47e3-b41d-39b608260e69\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") "
Oct 31 08:06:12.894604 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894606    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-certs\") pod \"c690eb53-1c5d-47e3-b41d-39b608260e69\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") "
Oct 31 08:06:12.894971 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894632    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-envoy\") pod \"c690eb53-1c5d-47e3-b41d-39b608260e69\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") "
Oct 31 08:06:12.894971 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894674    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-token\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-token\") pod \"c690eb53-1c5d-47e3-b41d-39b608260e69\" (UID: \"c690eb53-1c5d-47e3-b41d-39b608260e69\") "
Oct 31 08:06:12.894971 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.894819    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-data" pod "c690eb53-1c5d-47e3-b41d-39b608260e69" (UID: "c690eb53-1c5d-47e3-b41d-39b608260e69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:12.895134 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.895075    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:12.895946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.895317    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-credential-socket" pod "c690eb53-1c5d-47e3-b41d-39b608260e69" (UID: "c690eb53-1c5d-47e3-b41d-39b608260e69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:12.895946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.895630    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c690eb53-1c5d-47e3-b41d-39b608260e69-istiod-ca-cert" pod "c690eb53-1c5d-47e3-b41d-39b608260e69" (UID: "c690eb53-1c5d-47e3-b41d-39b608260e69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:06:12.895946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.895676    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-socket" pod "c690eb53-1c5d-47e3-b41d-39b608260e69" (UID: "c690eb53-1c5d-47e3-b41d-39b608260e69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:12.895946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.895893    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-certs" pod "c690eb53-1c5d-47e3-b41d-39b608260e69" (UID: "c690eb53-1c5d-47e3-b41d-39b608260e69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:12.897579 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.897518    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-kube-api-access-zkb8r" pod "c690eb53-1c5d-47e3-b41d-39b608260e69" (UID: "c690eb53-1c5d-47e3-b41d-39b608260e69"). InnerVolumeSpecName "kube-api-access-zkb8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:06:12.897712 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.897650    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-token" pod "c690eb53-1c5d-47e3-b41d-39b608260e69" (UID: "c690eb53-1c5d-47e3-b41d-39b608260e69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:06:12.898152 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.898119    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-podinfo" pod "c690eb53-1c5d-47e3-b41d-39b608260e69" (UID: "c690eb53-1c5d-47e3-b41d-39b608260e69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 08:06:12.899119 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.899094    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-envoy" pod "c690eb53-1c5d-47e3-b41d-39b608260e69" (UID: "c690eb53-1c5d-47e3-b41d-39b608260e69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:06:12.995975 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.995877    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:12.995975 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.995914    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:12.995975 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.995927    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:12.995975 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.995937    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c690eb53-1c5d-47e3-b41d-39b608260e69-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:12.995975 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.995945    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkb8r\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-kube-api-access-zkb8r\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:12.995975 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.995954    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:12.995975 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.995963    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:12.995975 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:12.995970    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c690eb53-1c5d-47e3-b41d-39b608260e69-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:13.068239 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.068203    5829 generic.go:356] "Generic (PLEG): container finished" podID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerID="8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468" exitCode=0
Oct 31 08:06:13.068463 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.068258    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"
Oct 31 08:06:13.068463 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.068323    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" event={"ID":"c690eb53-1c5d-47e3-b41d-39b608260e69","Type":"ContainerDied","Data":"8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468"}
Oct 31 08:06:13.068463 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.068399    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b" event={"ID":"c690eb53-1c5d-47e3-b41d-39b608260e69","Type":"ContainerDied","Data":"bd5cdd2899ecbd4e36d9f9468493bc43f87f6b06d3d16ff671f227a0973166b8"}
Oct 31 08:06:13.068463 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.068428    5829 scope.go:122] "RemoveContainer" containerID="8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468"
Oct 31 08:06:13.078541 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.078519    5829 scope.go:122] "RemoveContainer" containerID="66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18"
Oct 31 08:06:13.088043 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.088023    5829 scope.go:122] "RemoveContainer" containerID="8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468"
Oct 31 08:06:13.088343 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:06:13.088321    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468\": container with ID starting with 8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468 not found: ID does not exist" containerID="8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468"
Oct 31 08:06:13.088428 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.088359    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468"} err="failed to get container status \"8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468\": rpc error: code = NotFound desc = could not find container \"8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468\": container with ID starting with 8156a8cd5018496352ac933774ef10549f3e696e6ed08ce21d8277459b1b9468 not found: ID does not exist"
Oct 31 08:06:13.088428 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.088385    5829 scope.go:122] "RemoveContainer" containerID="66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18"
Oct 31 08:06:13.088648 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:06:13.088632    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18\": container with ID starting with 66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18 not found: ID does not exist" containerID="66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18"
Oct 31 08:06:13.088685 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.088656    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18"} err="failed to get container status \"66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18\": rpc error: code = NotFound desc = could not find container \"66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18\": container with ID starting with 66ad8b2434624668f3eaab0a0d95765502a0db0a1c7da3c4063e7f1f7f5e4c18 not found: ID does not exist"
Oct 31 08:06:13.120799 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.120765    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"]
Oct 31 08:06:13.123828 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.123797    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-0503df-openshift-default-59cb94669-sdq6b"]
Oct 31 08:06:13.973215 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:13.973183    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" path="/var/lib/kubelet/pods/c690eb53-1c5d-47e3-b41d-39b608260e69/volumes"
Oct 31 08:06:16.509542 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:06:16.509499    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:06:27.979103 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:27.979061    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:06:34.720829 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.720793    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"]
Oct 31 08:06:34.721246 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721220    5829 cpu_manager.go:414] "containerMap: removing container" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy"
Oct 31 08:06:34.721246 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721231    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy"
Oct 31 08:06:34.721347 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721254    5829 cpu_manager.go:414] "containerMap: removing container" podUID="d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13" containerName="revoke-keys"
Oct 31 08:06:34.721347 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721260    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13" containerName="revoke-keys"
Oct 31 08:06:34.721347 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721274    5829 cpu_manager.go:414] "containerMap: removing container" podUID="6362a4d9-817a-418f-a08d-1219d825dc73" containerName="payload-processing"
Oct 31 08:06:34.721347 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721294    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6362a4d9-817a-418f-a08d-1219d825dc73" containerName="payload-processing"
Oct 31 08:06:34.721347 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721307    5829 cpu_manager.go:414] "containerMap: removing container" podUID="6445859f-f36d-4fac-93e5-153838394b00" containerName="payload-pre-processing"
Oct 31 08:06:34.721347 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721312    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6445859f-f36d-4fac-93e5-153838394b00" containerName="payload-pre-processing"
Oct 31 08:06:34.721535 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721397    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy"
Oct 31 08:06:34.721535 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721407    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c624b3-cbe6-4ea3-86b4-e5ac8f6c7d13" containerName="revoke-keys"
Oct 31 08:06:34.721535 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721415    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="6362a4d9-817a-418f-a08d-1219d825dc73" containerName="payload-processing"
Oct 31 08:06:34.721535 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.721422    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="6445859f-f36d-4fac-93e5-153838394b00" containerName="payload-pre-processing"
Oct 31 08:06:34.724853 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.724836    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:34.743498 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.743468    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"]
Oct 31 08:06:34.754318 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.754275    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:06:34.754519 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.754325    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 08:06:34.754595 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.754418    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:06:34.754595 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.754507    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:06:34.842089 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.842048    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nww2x\" (UniqueName: \"kubernetes.io/projected/46935671-340a-4f4a-b63c-9690b43c81b0-kube-api-access-nww2x\") pod \"maas-api-revoke-sub-f03629db-p4t2z\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:34.842273 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.842103    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-p4t2z\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:34.842360 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.842264    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-p4t2z\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:34.943078 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.943036    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-p4t2z\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:34.943271 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.943143    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-p4t2z\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:34.943271 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.943212    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nww2x\" (UniqueName: \"kubernetes.io/projected/46935671-340a-4f4a-b63c-9690b43c81b0-kube-api-access-nww2x\") pod \"maas-api-revoke-sub-f03629db-p4t2z\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:34.943937 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.943901    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-p4t2z\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:34.945513 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.945489    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-p4t2z\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:34.963160 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:34.963132    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nww2x\" (UniqueName: \"kubernetes.io/projected/46935671-340a-4f4a-b63c-9690b43c81b0-kube-api-access-nww2x\") pod \"maas-api-revoke-sub-f03629db-p4t2z\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:35.041250 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:35.041166    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:35.341386 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:35.341360    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"]
Oct 31 08:06:35.344247 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:06:35.344218    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46935671_340a_4f4a_b63c_9690b43c81b0.slice/crio-101ed1bea272a6b5d4741931d5b6a827dd697c45f5d10aa81241cea2b9a1585b WatchSource:0}: Error finding container 101ed1bea272a6b5d4741931d5b6a827dd697c45f5d10aa81241cea2b9a1585b: Status 404 returned error can't find the container with id 101ed1bea272a6b5d4741931d5b6a827dd697c45f5d10aa81241cea2b9a1585b
Oct 31 08:06:36.214156 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:36.214123    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z" event={"ID":"46935671-340a-4f4a-b63c-9690b43c81b0","Type":"ContainerStarted","Data":"49c2643cfd211a9335b9363a070d62ed46a4243174935115066ac186ca23df50"}
Oct 31 08:06:36.214555 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:36.214195    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z" event={"ID":"46935671-340a-4f4a-b63c-9690b43c81b0","Type":"ContainerStarted","Data":"101ed1bea272a6b5d4741931d5b6a827dd697c45f5d10aa81241cea2b9a1585b"}
Oct 31 08:06:37.025532 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.025494    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"]
Oct 31 08:06:37.038571 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.038538    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-hpqcq"]
Oct 31 08:06:37.221395 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.221358    5829 generic.go:356] "Generic (PLEG): container finished" podID="46935671-340a-4f4a-b63c-9690b43c81b0" containerID="49c2643cfd211a9335b9363a070d62ed46a4243174935115066ac186ca23df50" exitCode=0
Oct 31 08:06:37.221806 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.221451    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z" event={"ID":"46935671-340a-4f4a-b63c-9690b43c81b0","Type":"ContainerDied","Data":"49c2643cfd211a9335b9363a070d62ed46a4243174935115066ac186ca23df50"}
Oct 31 08:06:37.361118 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.361092    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:37.470198 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.470163    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-xks-ca\") pod \"46935671-340a-4f4a-b63c-9690b43c81b0\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") "
Oct 31 08:06:37.470423 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.470265    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46935671-340a-4f4a-b63c-9690b43c81b0-kube-api-access-nww2x\" (UniqueName: \"kubernetes.io/projected/46935671-340a-4f4a-b63c-9690b43c81b0-kube-api-access-nww2x\") pod \"46935671-340a-4f4a-b63c-9690b43c81b0\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") "
Oct 31 08:06:37.470423 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.470324    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-openshift-ca\") pod \"46935671-340a-4f4a-b63c-9690b43c81b0\" (UID: \"46935671-340a-4f4a-b63c-9690b43c81b0\") "
Oct 31 08:06:37.470686 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.470661    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-openshift-ca" pod "46935671-340a-4f4a-b63c-9690b43c81b0" (UID: "46935671-340a-4f4a-b63c-9690b43c81b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:06:37.470764 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.470752    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:37.472152 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.472130    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-xks-ca" pod "46935671-340a-4f4a-b63c-9690b43c81b0" (UID: "46935671-340a-4f4a-b63c-9690b43c81b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:06:37.472254 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.472232    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46935671-340a-4f4a-b63c-9690b43c81b0-kube-api-access-nww2x" pod "46935671-340a-4f4a-b63c-9690b43c81b0" (UID: "46935671-340a-4f4a-b63c-9690b43c81b0"). InnerVolumeSpecName "kube-api-access-nww2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:06:37.571579 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.571545    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nww2x\" (UniqueName: \"kubernetes.io/projected/46935671-340a-4f4a-b63c-9690b43c81b0-kube-api-access-nww2x\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:37.571579 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.571579    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46935671-340a-4f4a-b63c-9690b43c81b0-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:06:37.973180 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:37.973095    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b1508e-8c84-4c8e-9a0c-acc65b3089f1" path="/var/lib/kubelet/pods/b0b1508e-8c84-4c8e-9a0c-acc65b3089f1/volumes"
Oct 31 08:06:38.227525 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:38.227434    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z" event={"ID":"46935671-340a-4f4a-b63c-9690b43c81b0","Type":"ContainerDied","Data":"101ed1bea272a6b5d4741931d5b6a827dd697c45f5d10aa81241cea2b9a1585b"}
Oct 31 08:06:38.227525 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:38.227459    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"
Oct 31 08:06:38.227525 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:38.227494    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="101ed1bea272a6b5d4741931d5b6a827dd697c45f5d10aa81241cea2b9a1585b"
Oct 31 08:06:45.937796 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:06:45.937763    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:06:46.510615 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:06:46.510570    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:07:00.270511 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.270461    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"]
Oct 31 08:07:00.271074 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.270891    5829 cpu_manager.go:414] "containerMap: removing container" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy"
Oct 31 08:07:00.271074 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.270903    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy"
Oct 31 08:07:00.271074 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.270947    5829 cpu_manager.go:414] "containerMap: removing container" podUID="46935671-340a-4f4a-b63c-9690b43c81b0" containerName="revoke-keys"
Oct 31 08:07:00.271074 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.270953    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46935671-340a-4f4a-b63c-9690b43c81b0" containerName="revoke-keys"
Oct 31 08:07:00.271074 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.271014    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="c690eb53-1c5d-47e3-b41d-39b608260e69" containerName="istio-proxy"
Oct 31 08:07:00.271074 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.271030    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="46935671-340a-4f4a-b63c-9690b43c81b0" containerName="revoke-keys"
Oct 31 08:07:00.275740 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.275717    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.293689 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.293655    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"]
Oct 31 08:07:00.304981 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.304216    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:07:00.304981 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.304268    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 08:07:00.304981 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.304313    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:07:00.304981 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.304320    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:07:00.329652 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.329620    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z9vwr\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.329857 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.329781    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6kb2\" (UniqueName: \"kubernetes.io/projected/97110768-e5d1-4b30-bf17-c3a38dad0e58-kube-api-access-x6kb2\") pod \"maas-api-revoke-sub-0165d7d2-z9vwr\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.329857 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.329825    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z9vwr\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.430991 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.430950    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6kb2\" (UniqueName: \"kubernetes.io/projected/97110768-e5d1-4b30-bf17-c3a38dad0e58-kube-api-access-x6kb2\") pod \"maas-api-revoke-sub-0165d7d2-z9vwr\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.430991 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.430997    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z9vwr\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.431264 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.431045    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z9vwr\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.431694 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.431671    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-z9vwr\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.433631 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.433599    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-z9vwr\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.450675 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.450644    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6kb2\" (UniqueName: \"kubernetes.io/projected/97110768-e5d1-4b30-bf17-c3a38dad0e58-kube-api-access-x6kb2\") pod \"maas-api-revoke-sub-0165d7d2-z9vwr\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.589532 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.589442    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:00.887132 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:00.887103    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"]
Oct 31 08:07:00.890830 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:07:00.890793    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97110768_e5d1_4b30_bf17_c3a38dad0e58.slice/crio-dc050bf09ad420ced381e42a0d87797451ea17f37fabf2b7b2f95eebf403c481 WatchSource:0}: Error finding container dc050bf09ad420ced381e42a0d87797451ea17f37fabf2b7b2f95eebf403c481: Status 404 returned error can't find the container with id dc050bf09ad420ced381e42a0d87797451ea17f37fabf2b7b2f95eebf403c481
Oct 31 08:07:01.358961 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:01.358916    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr" event={"ID":"97110768-e5d1-4b30-bf17-c3a38dad0e58","Type":"ContainerStarted","Data":"6fcd4bd65bb93a5c23fe43c6481132837e443c55638fbec858d7bc5db9be4771"}
Oct 31 08:07:01.359392 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:01.358986    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr" event={"ID":"97110768-e5d1-4b30-bf17-c3a38dad0e58","Type":"ContainerStarted","Data":"dc050bf09ad420ced381e42a0d87797451ea17f37fabf2b7b2f95eebf403c481"}
Oct 31 08:07:01.398270 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:01.398219    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr" podStartSLOduration=1.398204986 podStartE2EDuration="1.398204986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:07:01.396908853 +0000 UTC m=+2285.536069440" watchObservedRunningTime="2026-10-31 08:07:01.398204986 +0000 UTC m=+2285.537365598"
Oct 31 08:07:02.366993 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:02.366957    5829 generic.go:356] "Generic (PLEG): container finished" podID="97110768-e5d1-4b30-bf17-c3a38dad0e58" containerID="6fcd4bd65bb93a5c23fe43c6481132837e443c55638fbec858d7bc5db9be4771" exitCode=0
Oct 31 08:07:02.367416 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:02.367040    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr" event={"ID":"97110768-e5d1-4b30-bf17-c3a38dad0e58","Type":"ContainerDied","Data":"6fcd4bd65bb93a5c23fe43c6481132837e443c55638fbec858d7bc5db9be4771"}
Oct 31 08:07:03.508376 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.508345    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:03.563545 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.563513    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-xks-ca\") pod \"97110768-e5d1-4b30-bf17-c3a38dad0e58\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") "
Oct 31 08:07:03.563743 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.563610    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97110768-e5d1-4b30-bf17-c3a38dad0e58-kube-api-access-x6kb2\" (UniqueName: \"kubernetes.io/projected/97110768-e5d1-4b30-bf17-c3a38dad0e58-kube-api-access-x6kb2\") pod \"97110768-e5d1-4b30-bf17-c3a38dad0e58\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") "
Oct 31 08:07:03.563743 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.563709    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-openshift-ca\") pod \"97110768-e5d1-4b30-bf17-c3a38dad0e58\" (UID: \"97110768-e5d1-4b30-bf17-c3a38dad0e58\") "
Oct 31 08:07:03.564090 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.564063    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-openshift-ca" pod "97110768-e5d1-4b30-bf17-c3a38dad0e58" (UID: "97110768-e5d1-4b30-bf17-c3a38dad0e58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:03.565655 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.565624    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-xks-ca" pod "97110768-e5d1-4b30-bf17-c3a38dad0e58" (UID: "97110768-e5d1-4b30-bf17-c3a38dad0e58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:07:03.565763 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.565679    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97110768-e5d1-4b30-bf17-c3a38dad0e58-kube-api-access-x6kb2" pod "97110768-e5d1-4b30-bf17-c3a38dad0e58" (UID: "97110768-e5d1-4b30-bf17-c3a38dad0e58"). InnerVolumeSpecName "kube-api-access-x6kb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:07:03.665022 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.664978    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:07:03.665022 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.665015    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6kb2\" (UniqueName: \"kubernetes.io/projected/97110768-e5d1-4b30-bf17-c3a38dad0e58-kube-api-access-x6kb2\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:07:03.665022 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:03.665026    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97110768-e5d1-4b30-bf17-c3a38dad0e58-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:07:04.378224 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:04.378133    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"
Oct 31 08:07:04.378224 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:04.378171    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr" event={"ID":"97110768-e5d1-4b30-bf17-c3a38dad0e58","Type":"ContainerDied","Data":"dc050bf09ad420ced381e42a0d87797451ea17f37fabf2b7b2f95eebf403c481"}
Oct 31 08:07:04.378224 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:04.378222    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc050bf09ad420ced381e42a0d87797451ea17f37fabf2b7b2f95eebf403c481"
Oct 31 08:07:08.026890 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:08.026858    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"]
Oct 31 08:07:08.040161 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:08.040128    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157e87f5-2jtd9"]
Oct 31 08:07:09.973569 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:09.973533    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd9045a-c75c-472c-97cf-e5d73b104470" path="/var/lib/kubelet/pods/6cd9045a-c75c-472c-97cf-e5d73b104470/volumes"
Oct 31 08:07:11.161089 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.161057    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"]
Oct 31 08:07:11.161517 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.161507    5829 cpu_manager.go:414] "containerMap: removing container" podUID="97110768-e5d1-4b30-bf17-c3a38dad0e58" containerName="revoke-keys"
Oct 31 08:07:11.161560 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.161519    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97110768-e5d1-4b30-bf17-c3a38dad0e58" containerName="revoke-keys"
Oct 31 08:07:11.161626 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.161615    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="97110768-e5d1-4b30-bf17-c3a38dad0e58" containerName="revoke-keys"
Oct 31 08:07:11.166127 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.166110    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.182077 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.182030    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:07:11.195135 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.195104    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 08:07:11.195135 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.195109    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:07:11.195395 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.195333    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:07:11.196988 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.196967    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"]
Oct 31 08:07:11.240249 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.240207    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nfrp9\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.240249 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.240258    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24bpf\" (UniqueName: \"kubernetes.io/projected/7e45b293-03d2-4957-b6b4-c52b880e696a-kube-api-access-24bpf\") pod \"maas-api-revoke-sub-6b19338f-nfrp9\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.240541 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.240357    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nfrp9\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.341745 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.341705    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nfrp9\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.341965 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.341774    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24bpf\" (UniqueName: \"kubernetes.io/projected/7e45b293-03d2-4957-b6b4-c52b880e696a-kube-api-access-24bpf\") pod \"maas-api-revoke-sub-6b19338f-nfrp9\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.341965 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.341819    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nfrp9\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.342531 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.342504    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-nfrp9\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.344319 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.344294    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-nfrp9\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.361817 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.361785    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24bpf\" (UniqueName: \"kubernetes.io/projected/7e45b293-03d2-4957-b6b4-c52b880e696a-kube-api-access-24bpf\") pod \"maas-api-revoke-sub-6b19338f-nfrp9\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.478888 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.478801    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:11.783732 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:11.783689    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"]
Oct 31 08:07:11.786194 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:07:11.786163    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e45b293_03d2_4957_b6b4_c52b880e696a.slice/crio-5f49dfbc62cf43bdffaf716067ccfc0e3542682a559b20a4c3bcd81076ff08e9 WatchSource:0}: Error finding container 5f49dfbc62cf43bdffaf716067ccfc0e3542682a559b20a4c3bcd81076ff08e9: Status 404 returned error can't find the container with id 5f49dfbc62cf43bdffaf716067ccfc0e3542682a559b20a4c3bcd81076ff08e9
Oct 31 08:07:12.426333 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:12.426298    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9" event={"ID":"7e45b293-03d2-4957-b6b4-c52b880e696a","Type":"ContainerStarted","Data":"2ecba108863ed684e24b55c45cd58e75e2b6b8b22d94a213ce2c2f6a469fcd97"}
Oct 31 08:07:12.426778 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:12.426364    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9" event={"ID":"7e45b293-03d2-4957-b6b4-c52b880e696a","Type":"ContainerStarted","Data":"5f49dfbc62cf43bdffaf716067ccfc0e3542682a559b20a4c3bcd81076ff08e9"}
Oct 31 08:07:12.465846 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:12.465783    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9" podStartSLOduration=1.465769445 podStartE2EDuration="1.465769445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 08:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:07:12.464516495 +0000 UTC m=+2296.603677093" watchObservedRunningTime="2026-10-31 08:07:12.465769445 +0000 UTC m=+2296.604930030"
Oct 31 08:07:13.433696 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:13.433663    5829 generic.go:356] "Generic (PLEG): container finished" podID="7e45b293-03d2-4957-b6b4-c52b880e696a" containerID="2ecba108863ed684e24b55c45cd58e75e2b6b8b22d94a213ce2c2f6a469fcd97" exitCode=0
Oct 31 08:07:13.434116 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:13.433702    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9" event={"ID":"7e45b293-03d2-4957-b6b4-c52b880e696a","Type":"ContainerDied","Data":"2ecba108863ed684e24b55c45cd58e75e2b6b8b22d94a213ce2c2f6a469fcd97"}
Oct 31 08:07:14.582973 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.582946    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:14.678751 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.678712    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-openshift-ca\") pod \"7e45b293-03d2-4957-b6b4-c52b880e696a\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") "
Oct 31 08:07:14.678751 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.678751    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e45b293-03d2-4957-b6b4-c52b880e696a-kube-api-access-24bpf\" (UniqueName: \"kubernetes.io/projected/7e45b293-03d2-4957-b6b4-c52b880e696a-kube-api-access-24bpf\") pod \"7e45b293-03d2-4957-b6b4-c52b880e696a\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") "
Oct 31 08:07:14.679027 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.678801    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-xks-ca\") pod \"7e45b293-03d2-4957-b6b4-c52b880e696a\" (UID: \"7e45b293-03d2-4957-b6b4-c52b880e696a\") "
Oct 31 08:07:14.679127 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.679100    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-openshift-ca" pod "7e45b293-03d2-4957-b6b4-c52b880e696a" (UID: "7e45b293-03d2-4957-b6b4-c52b880e696a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:07:14.681166 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.681134    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e45b293-03d2-4957-b6b4-c52b880e696a-kube-api-access-24bpf" pod "7e45b293-03d2-4957-b6b4-c52b880e696a" (UID: "7e45b293-03d2-4957-b6b4-c52b880e696a"). InnerVolumeSpecName "kube-api-access-24bpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:07:14.681166 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.681145    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-xks-ca" pod "7e45b293-03d2-4957-b6b4-c52b880e696a" (UID: "7e45b293-03d2-4957-b6b4-c52b880e696a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 08:07:14.780456 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.780375    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-openshift-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:07:14.780456 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.780403    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24bpf\" (UniqueName: \"kubernetes.io/projected/7e45b293-03d2-4957-b6b4-c52b880e696a-kube-api-access-24bpf\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:07:14.780456 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:14.780414    5829 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e45b293-03d2-4957-b6b4-c52b880e696a-maas-api-xks-ca\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:07:15.449645 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:15.449613    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"
Oct 31 08:07:15.449645 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:15.449632    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9" event={"ID":"7e45b293-03d2-4957-b6b4-c52b880e696a","Type":"ContainerDied","Data":"5f49dfbc62cf43bdffaf716067ccfc0e3542682a559b20a4c3bcd81076ff08e9"}
Oct 31 08:07:15.449871 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:15.449676    5829 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f49dfbc62cf43bdffaf716067ccfc0e3542682a559b20a4c3bcd81076ff08e9"
Oct 31 08:07:16.511216 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:07:16.511174    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:07:22.354553 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:22.354511    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-p6h7j"]
Oct 31 08:07:46.510757 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:07:46.510717    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:07:52.860733 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:52.860701    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-66fd44d76c-d9rzk_56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5/manager/0.log"
Oct 31 08:07:59.015719 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:59.015685    5829 scope.go:122] "RemoveContainer" containerID="d095be5b34a65b26d28e6ecf3a0b00f883431fc31191a5995482c926713d2179"
Oct 31 08:07:59.027126 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:07:59.027105    5829 scope.go:122] "RemoveContainer" containerID="cadf2db108dd2600d8d80f562b2c0e35cf7bb7f3f467d0ee9368acc3bca1c51a"
Oct 31 08:08:05.716305 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:05.716260    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-z9vwr_97110768-e5d1-4b30-bf17-c3a38dad0e58/revoke-keys/0.log"
Oct 31 08:08:05.888072 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:05.888038    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nfrp9_7e45b293-03d2-4957-b6b4-c52b880e696a/revoke-keys/0.log"
Oct 31 08:08:06.070118 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:06.070030    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-p4t2z_46935671-340a-4f4a-b63c-9690b43c81b0/revoke-keys/0.log"
Oct 31 08:08:16.511638 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:08:16.511531    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:08:18.293261 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:18.293229    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kdptv_565e1131-e783-4a77-8b24-71d6349f545e/kuadrant-console-plugin/0.log"
Oct 31 08:08:18.739266 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:18.739234    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p6h7j_ce64b388-ff7c-4cc1-8bea-74e05c0cfd97/limitador/0.log"
Oct 31 08:08:30.307441 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:30.307405    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cdbcb4849-t7xbb_3992d138-a7d9-42f6-b3b7-6dc9e2cdb478/kube-auth-proxy/0.log"
Oct 31 08:08:31.097685 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:31.097654    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-hkpt2_305fd353-98c5-4686-b2b2-b7ee1a752d87/istio-proxy/0.log"
Oct 31 08:08:31.491339 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:31.491304    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-m8njr_a49dc456-e746-4d51-94ab-124f8abeeb49/istio-proxy/0.log"
Oct 31 08:08:32.248968 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:32.248932    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-vt92f_2991db1f-d2e3-428f-8746-ec01ace2e05b/istio-proxy/0.log"
Oct 31 08:08:32.617491 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:32.617397    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rw25z_2b71dae3-c259-4a13-af86-ab549638b3d8/payload-pre-processing/0.log"
Oct 31 08:08:32.798275 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:32.798242    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sp8xc_02eee9ac-9709-4b6b-a26c-3fd5ea5e1658/payload-processing/0.log"
Oct 31 08:08:34.301874 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:34.301839    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-b6pmz_bed58086-ad9a-45b7-a859-65f197560321/main/0.log"
Oct 31 08:08:34.864095 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:34.864062    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-jm2hw_cdcc4524-8f99-4e74-8369-90b0b7ca86d5/main/0.log"
Oct 31 08:08:35.053471 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:35.053436    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6_9271b52d-799d-4bd9-bf35-3c0b466b9346/main/0.log"
Oct 31 08:08:35.244534 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:35.244492    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4_0b5c0eeb-8829-4fa6-8ce0-617cbf88b831/main/0.log"
Oct 31 08:08:46.510318 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:08:46.510252    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:08:58.232858 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:58.232822    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 08:08:58.237355 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:58.237332    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 08:08:59.052751 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:08:59.052718    5829 scope.go:122] "RemoveContainer" containerID="77c375297c1b2a14bb7f04554b101606ffb640abdb80574e84a8c41c6e76e264"
Oct 31 08:09:15.098635 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:15.098596    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-66fd44d76c-d9rzk_56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5/manager/0.log"
Oct 31 08:09:16.510862 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:09:16.510827    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:09:27.565200 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:27.565169    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-z9vwr_97110768-e5d1-4b30-bf17-c3a38dad0e58/revoke-keys/0.log"
Oct 31 08:09:27.745823 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:27.745788    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-nfrp9_7e45b293-03d2-4957-b6b4-c52b880e696a/revoke-keys/0.log"
Oct 31 08:09:37.916916 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:37.916832    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-p4t2z_46935671-340a-4f4a-b63c-9690b43c81b0/revoke-keys/0.log"
Oct 31 08:09:39.836338 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:39.836309    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kdptv_565e1131-e783-4a77-8b24-71d6349f545e/kuadrant-console-plugin/0.log"
Oct 31 08:09:40.249031 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:40.248996    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p6h7j_ce64b388-ff7c-4cc1-8bea-74e05c0cfd97/limitador/0.log"
Oct 31 08:09:41.560014 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:41.559981    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cdbcb4849-t7xbb_3992d138-a7d9-42f6-b3b7-6dc9e2cdb478/kube-auth-proxy/0.log"
Oct 31 08:09:42.305183 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:42.305153    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-hkpt2_305fd353-98c5-4686-b2b2-b7ee1a752d87/istio-proxy/0.log"
Oct 31 08:09:42.687695 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:42.687655    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-m8njr_a49dc456-e746-4d51-94ab-124f8abeeb49/istio-proxy/0.log"
Oct 31 08:09:43.430833 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:43.430801    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-vt92f_2991db1f-d2e3-428f-8746-ec01ace2e05b/istio-proxy/0.log"
Oct 31 08:09:43.786741 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:43.786655    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rw25z_2b71dae3-c259-4a13-af86-ab549638b3d8/payload-pre-processing/0.log"
Oct 31 08:09:43.976864 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:43.976832    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sp8xc_02eee9ac-9709-4b6b-a26c-3fd5ea5e1658/payload-processing/0.log"
Oct 31 08:09:45.061124 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:45.061092    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-b6pmz_bed58086-ad9a-45b7-a859-65f197560321/main/0.log"
Oct 31 08:09:45.633252 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:45.633222    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-jm2hw_cdcc4524-8f99-4e74-8369-90b0b7ca86d5/main/0.log"
Oct 31 08:09:45.823702 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:45.823666    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-hgfk6_9271b52d-799d-4bd9-bf35-3c0b466b9346/main/0.log"
Oct 31 08:09:46.019257 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:46.019227    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-ghgx4_0b5c0eeb-8829-4fa6-8ce0-617cbf88b831/main/0.log"
Oct 31 08:09:46.510398 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:09:46.510361    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:09:59.076915 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:09:59.076875    5829 scope.go:122] "RemoveContainer" containerID="0e136b5499a218be9a4a890cae9554606ec66bc34aa70ab81da0d48632ab5a77"
Oct 31 08:10:16.510755 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:10:16.510719    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:10:46.510857 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:10:46.510817    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:11:16.510733 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:11:16.510649    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:11:38.027767 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:11:38.027729    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"]
Oct 31 08:11:38.040481 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:11:38.040441    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-p4t2z"]
Oct 31 08:11:39.973553 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:11:39.973513    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46935671-340a-4f4a-b63c-9690b43c81b0" path="/var/lib/kubelet/pods/46935671-340a-4f4a-b63c-9690b43c81b0/volumes"
Oct 31 08:11:43.905392 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:11:43.905353    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sp8xc_02eee9ac-9709-4b6b-a26c-3fd5ea5e1658/payload-processing/0.log"
Oct 31 08:11:44.099630 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:11:44.099601    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sp8xc_02eee9ac-9709-4b6b-a26c-3fd5ea5e1658/payload-processing/0.log"
Oct 31 08:11:46.510638 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:11:46.510595    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:11:59.106320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:11:59.106267    5829 scope.go:122] "RemoveContainer" containerID="fad1eb3c718b728310ea7d5a10ac7245889afd0375d7efdca7906333bbd73e39"
Oct 31 08:12:04.025062 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:04.025026    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"]
Oct 31 08:12:04.037950 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:04.037916    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-z9vwr"]
Oct 31 08:12:05.973348 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:05.973313    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97110768-e5d1-4b30-bf17-c3a38dad0e58" path="/var/lib/kubelet/pods/97110768-e5d1-4b30-bf17-c3a38dad0e58/volumes"
Oct 31 08:12:15.026378 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:15.026336    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"]
Oct 31 08:12:15.040099 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:15.040062    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-nfrp9"]
Oct 31 08:12:15.974075 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:15.974039    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e45b293-03d2-4957-b6b4-c52b880e696a" path="/var/lib/kubelet/pods/7e45b293-03d2-4957-b6b4-c52b880e696a/volumes"
Oct 31 08:12:16.510322 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:12:16.510261    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:12:46.307695 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:46.307579    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"]
Oct 31 08:12:46.308269 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:46.307805    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy" containerID="cri-o://dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c" gracePeriod=30
Oct 31 08:12:46.320872 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:46.320812    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"]
Oct 31 08:12:46.321080 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:46.321031    5829 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy" containerID="cri-o://fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6" gracePeriod=30
Oct 31 08:12:46.510535 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:12:46.510496    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:12:51.587026 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.586998    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 08:12:51.609552 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.609527    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 08:12:51.649144 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.649108    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-token\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-token\") pod \"2991db1f-d2e3-428f-8746-ec01ace2e05b\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") "
Oct 31 08:12:51.649351 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.649166    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-certs\") pod \"2991db1f-d2e3-428f-8746-ec01ace2e05b\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") "
Oct 31 08:12:51.649351 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.649208    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-podinfo\") pod \"2991db1f-d2e3-428f-8746-ec01ace2e05b\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") "
Oct 31 08:12:51.649593 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.649546    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-certs" pod "2991db1f-d2e3-428f-8746-ec01ace2e05b" (UID: "2991db1f-d2e3-428f-8746-ec01ace2e05b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.649785 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.649763    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-socket\") pod \"2991db1f-d2e3-428f-8746-ec01ace2e05b\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") "
Oct 31 08:12:51.650429 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.649833    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-data\") pod \"2991db1f-d2e3-428f-8746-ec01ace2e05b\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") "
Oct 31 08:12:51.650429 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.649880    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2991db1f-d2e3-428f-8746-ec01ace2e05b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2991db1f-d2e3-428f-8746-ec01ace2e05b-istiod-ca-cert\") pod \"2991db1f-d2e3-428f-8746-ec01ace2e05b\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") "
Oct 31 08:12:51.650429 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.649948    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-kube-api-access-5p8p7\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-kube-api-access-5p8p7\") pod \"2991db1f-d2e3-428f-8746-ec01ace2e05b\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") "
Oct 31 08:12:51.650429 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.649992    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-credential-socket\") pod \"2991db1f-d2e3-428f-8746-ec01ace2e05b\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") "
Oct 31 08:12:51.650429 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.650024    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-envoy\") pod \"2991db1f-d2e3-428f-8746-ec01ace2e05b\" (UID: \"2991db1f-d2e3-428f-8746-ec01ace2e05b\") "
Oct 31 08:12:51.651271 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.650254    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-socket" pod "2991db1f-d2e3-428f-8746-ec01ace2e05b" (UID: "2991db1f-d2e3-428f-8746-ec01ace2e05b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.651415 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.651018    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-data" pod "2991db1f-d2e3-428f-8746-ec01ace2e05b" (UID: "2991db1f-d2e3-428f-8746-ec01ace2e05b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.651415 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.651181    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2991db1f-d2e3-428f-8746-ec01ace2e05b-istiod-ca-cert" pod "2991db1f-d2e3-428f-8746-ec01ace2e05b" (UID: "2991db1f-d2e3-428f-8746-ec01ace2e05b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:12:51.651415 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.651232    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-credential-socket" pod "2991db1f-d2e3-428f-8746-ec01ace2e05b" (UID: "2991db1f-d2e3-428f-8746-ec01ace2e05b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.651415 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.650823    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.653214 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.653169    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-podinfo" pod "2991db1f-d2e3-428f-8746-ec01ace2e05b" (UID: "2991db1f-d2e3-428f-8746-ec01ace2e05b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 08:12:51.653458 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.653177    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-token" pod "2991db1f-d2e3-428f-8746-ec01ace2e05b" (UID: "2991db1f-d2e3-428f-8746-ec01ace2e05b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:12:51.653562 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.653527    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-kube-api-access-5p8p7" pod "2991db1f-d2e3-428f-8746-ec01ace2e05b" (UID: "2991db1f-d2e3-428f-8746-ec01ace2e05b"). InnerVolumeSpecName "kube-api-access-5p8p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:12:51.653653 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.653629    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-envoy" pod "2991db1f-d2e3-428f-8746-ec01ace2e05b" (UID: "2991db1f-d2e3-428f-8746-ec01ace2e05b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.752325 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752228    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305fd353-98c5-4686-b2b2-b7ee1a752d87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305fd353-98c5-4686-b2b2-b7ee1a752d87-istiod-ca-cert\") pod \"305fd353-98c5-4686-b2b2-b7ee1a752d87\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") "
Oct 31 08:12:51.752577 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752402    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-credential-socket\") pod \"305fd353-98c5-4686-b2b2-b7ee1a752d87\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") "
Oct 31 08:12:51.752577 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752485    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-data\") pod \"305fd353-98c5-4686-b2b2-b7ee1a752d87\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") "
Oct 31 08:12:51.752577 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752514    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-token\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-token\") pod \"305fd353-98c5-4686-b2b2-b7ee1a752d87\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") "
Oct 31 08:12:51.752577 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752513    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305fd353-98c5-4686-b2b2-b7ee1a752d87-istiod-ca-cert" pod "305fd353-98c5-4686-b2b2-b7ee1a752d87" (UID: "305fd353-98c5-4686-b2b2-b7ee1a752d87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 08:12:51.752577 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752545    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-socket\") pod \"305fd353-98c5-4686-b2b2-b7ee1a752d87\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") "
Oct 31 08:12:51.752898 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752607    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-envoy\") pod \"305fd353-98c5-4686-b2b2-b7ee1a752d87\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") "
Oct 31 08:12:51.752898 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752658    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-certs\") pod \"305fd353-98c5-4686-b2b2-b7ee1a752d87\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") "
Oct 31 08:12:51.752898 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752703    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-podinfo\") pod \"305fd353-98c5-4686-b2b2-b7ee1a752d87\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") "
Oct 31 08:12:51.752898 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752731    5829 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-kube-api-access-97qvv\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-kube-api-access-97qvv\") pod \"305fd353-98c5-4686-b2b2-b7ee1a752d87\" (UID: \"305fd353-98c5-4686-b2b2-b7ee1a752d87\") "
Oct 31 08:12:51.752898 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.752823    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-data" pod "305fd353-98c5-4686-b2b2-b7ee1a752d87" (UID: "305fd353-98c5-4686-b2b2-b7ee1a752d87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753059    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-credential-socket" pod "305fd353-98c5-4686-b2b2-b7ee1a752d87" (UID: "305fd353-98c5-4686-b2b2-b7ee1a752d87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753075    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753089    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753100    5829 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-data\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753109    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2991db1f-d2e3-428f-8746-ec01ace2e05b-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753118    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8p7\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-kube-api-access-5p8p7\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753126    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753134    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753142    5829 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/305fd353-98c5-4686-b2b2-b7ee1a752d87-istiod-ca-cert\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753150    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753229 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753157    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2991db1f-d2e3-428f-8746-ec01ace2e05b-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.753810 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753500    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-certs" pod "305fd353-98c5-4686-b2b2-b7ee1a752d87" (UID: "305fd353-98c5-4686-b2b2-b7ee1a752d87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.753971 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.753931    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-socket" pod "305fd353-98c5-4686-b2b2-b7ee1a752d87" (UID: "305fd353-98c5-4686-b2b2-b7ee1a752d87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.755439 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.755410    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-envoy" pod "305fd353-98c5-4686-b2b2-b7ee1a752d87" (UID: "305fd353-98c5-4686-b2b2-b7ee1a752d87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 08:12:51.755851 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.755830    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-kube-api-access-97qvv" pod "305fd353-98c5-4686-b2b2-b7ee1a752d87" (UID: "305fd353-98c5-4686-b2b2-b7ee1a752d87"). InnerVolumeSpecName "kube-api-access-97qvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:12:51.755941 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.755834    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-podinfo" pod "305fd353-98c5-4686-b2b2-b7ee1a752d87" (UID: "305fd353-98c5-4686-b2b2-b7ee1a752d87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 08:12:51.756146 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.756122    5829 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-token" pod "305fd353-98c5-4686-b2b2-b7ee1a752d87" (UID: "305fd353-98c5-4686-b2b2-b7ee1a752d87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 08:12:51.854483 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.854389    5829 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-credential-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.854483 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.854425    5829 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-token\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.854483 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.854436    5829 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-socket\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.854483 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.854444    5829 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-envoy\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.854483 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.854453    5829 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/305fd353-98c5-4686-b2b2-b7ee1a752d87-workload-certs\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.854483 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.854461    5829 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/305fd353-98c5-4686-b2b2-b7ee1a752d87-istio-podinfo\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:51.854483 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:51.854471    5829 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97qvv\" (UniqueName: \"kubernetes.io/projected/305fd353-98c5-4686-b2b2-b7ee1a752d87-kube-api-access-97qvv\") on node \"ip-10-0-140-236.ec2.internal\" DevicePath \"\""
Oct 31 08:12:52.263903 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.263865    5829 generic.go:356] "Generic (PLEG): container finished" podID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerID="fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6" exitCode=0
Oct 31 08:12:52.264084 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.263913    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"
Oct 31 08:12:52.264084 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.263955    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" event={"ID":"305fd353-98c5-4686-b2b2-b7ee1a752d87","Type":"ContainerDied","Data":"fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6"}
Oct 31 08:12:52.264084 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.264017    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2" event={"ID":"305fd353-98c5-4686-b2b2-b7ee1a752d87","Type":"ContainerDied","Data":"9cb192a53408ccc274d45dffdd96768ece7ffa5267d415a998421430ca038447"}
Oct 31 08:12:52.264084 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.264047    5829 scope.go:122] "RemoveContainer" containerID="fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6"
Oct 31 08:12:52.265400 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.265372    5829 generic.go:356] "Generic (PLEG): container finished" podID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerID="dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c" exitCode=0
Oct 31 08:12:52.265530 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.265414    5829 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"
Oct 31 08:12:52.265530 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.265454    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" event={"ID":"2991db1f-d2e3-428f-8746-ec01ace2e05b","Type":"ContainerDied","Data":"dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c"}
Oct 31 08:12:52.265530 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.265506    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f" event={"ID":"2991db1f-d2e3-428f-8746-ec01ace2e05b","Type":"ContainerDied","Data":"a9d8ca6c5ea6c65cbb0dddb2449a8b8b3056226e2712c15b58dc53ab92262893"}
Oct 31 08:12:52.273472 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.273458    5829 scope.go:122] "RemoveContainer" containerID="fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6"
Oct 31 08:12:52.273768 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:12:52.273748    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6\": container with ID starting with fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6 not found: ID does not exist" containerID="fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6"
Oct 31 08:12:52.273812 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.273776    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6"} err="failed to get container status \"fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6\": rpc error: code = NotFound desc = could not find container \"fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6\": container with ID starting with fec5e7c79c79e0c80bb2ba9c5cfb52a26f04ffe3c468f08dff5e67f3807e45d6 not found: ID does not exist"
Oct 31 08:12:52.273812 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.273791    5829 scope.go:122] "RemoveContainer" containerID="dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c"
Oct 31 08:12:52.282168 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.282152    5829 scope.go:122] "RemoveContainer" containerID="dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c"
Oct 31 08:12:52.282502 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:12:52.282481    5829 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c\": container with ID starting with dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c not found: ID does not exist" containerID="dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c"
Oct 31 08:12:52.282607 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.282506    5829 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c"} err="failed to get container status \"dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c\": rpc error: code = NotFound desc = could not find container \"dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c\": container with ID starting with dcda2942f3f9f34bf64cb0f8fbf00227a048e2ebf272088c35083a5103cb389c not found: ID does not exist"
Oct 31 08:12:52.317592 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.317562    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"]
Oct 31 08:12:52.330429 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.330395    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-vt92f"]
Oct 31 08:12:52.356829 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.356801    5829 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"]
Oct 31 08:12:52.360451 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:52.360424    5829 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-645cf464d7-hkpt2"]
Oct 31 08:12:53.980233 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:53.974988    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" path="/var/lib/kubelet/pods/2991db1f-d2e3-428f-8746-ec01ace2e05b/volumes"
Oct 31 08:12:53.980233 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:53.975730    5829 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" path="/var/lib/kubelet/pods/305fd353-98c5-4686-b2b2-b7ee1a752d87/volumes"
Oct 31 08:12:59.127582 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:12:59.127541    5829 scope.go:122] "RemoveContainer" containerID="49c2643cfd211a9335b9363a070d62ed46a4243174935115066ac186ca23df50"
Oct 31 08:13:16.511118 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:13:16.511071    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:13:46.510932 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:13:46.510890    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:13:58.268122 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:13:58.268087    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 08:13:58.274131 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:13:58.274107    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 08:13:59.159783 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:13:59.159747    5829 scope.go:122] "RemoveContainer" containerID="2ecba108863ed684e24b55c45cd58e75e2b6b8b22d94a213ce2c2f6a469fcd97"
Oct 31 08:13:59.168739 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:13:59.168715    5829 scope.go:122] "RemoveContainer" containerID="6fcd4bd65bb93a5c23fe43c6481132837e443c55638fbec858d7bc5db9be4771"
Oct 31 08:14:10.009835 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:10.009750    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sp8xc_02eee9ac-9709-4b6b-a26c-3fd5ea5e1658/payload-processing/0.log"
Oct 31 08:14:10.190087 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:10.190051    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sp8xc_02eee9ac-9709-4b6b-a26c-3fd5ea5e1658/payload-processing/0.log"
Oct 31 08:14:14.748609 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:14.748568    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-82457_f0f4a456-26c2-4a45-bd30-b6795d5c03cc/global-pull-secret-syncer/0.log"
Oct 31 08:14:15.916272 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:15.916235    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pkj5d_132e4deb-32cc-4b7b-a179-44a9700f1bac/konnectivity-agent/0.log"
Oct 31 08:14:16.263610 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:16.263525    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-236.ec2.internal_e82d0dbde7394ec25631244e1f1d5cd9/haproxy/0.log"
Oct 31 08:14:16.510959 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:14:16.510921    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:14:22.372470 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:22.372444    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-kdptv_565e1131-e783-4a77-8b24-71d6349f545e/kuadrant-console-plugin/0.log"
Oct 31 08:14:22.625019 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:22.624926    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-p6h7j_ce64b388-ff7c-4cc1-8bea-74e05c0cfd97/limitador/0.log"
Oct 31 08:14:23.937445 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.937410    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"]
Oct 31 08:14:23.937878 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.937863    5829 cpu_manager.go:414] "containerMap: removing container" podUID="7e45b293-03d2-4957-b6b4-c52b880e696a" containerName="revoke-keys"
Oct 31 08:14:23.937938 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.937879    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e45b293-03d2-4957-b6b4-c52b880e696a" containerName="revoke-keys"
Oct 31 08:14:23.937938 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.937896    5829 cpu_manager.go:414] "containerMap: removing container" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy"
Oct 31 08:14:23.937938 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.937905    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy"
Oct 31 08:14:23.937938 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.937914    5829 cpu_manager.go:414] "containerMap: removing container" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy"
Oct 31 08:14:23.937938 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.937921    5829 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy"
Oct 31 08:14:23.938136 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.938060    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e45b293-03d2-4957-b6b4-c52b880e696a" containerName="revoke-keys"
Oct 31 08:14:23.938136 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.938076    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="2991db1f-d2e3-428f-8746-ec01ace2e05b" containerName="istio-proxy"
Oct 31 08:14:23.938136 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.938086    5829 memory_manager.go:365] "RemoveStaleState removing state" podUID="305fd353-98c5-4686-b2b2-b7ee1a752d87" containerName="istio-proxy"
Oct 31 08:14:23.942949 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.942925    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:23.959378 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:23.959341    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"]
Oct 31 08:14:24.037233 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.037200    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0fd0630-ea16-413c-a195-ff4cd58d8d21/alertmanager/0.log"
Oct 31 08:14:24.048857 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.048826    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pl6m9\"/\"kube-root-ca.crt\""
Oct 31 08:14:24.048857 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.048839    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pl6m9\"/\"openshift-service-ca.crt\""
Oct 31 08:14:24.049003 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.048939    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pl6m9\"/\"default-dockercfg-gs2j4\""
Oct 31 08:14:24.063344 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.063276    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-sys\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.063447 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.063397    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-lib-modules\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.063497 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.063459    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-proc\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.063557 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.063538    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-podres\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.063625 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.063609    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2v7l\" (UniqueName: \"kubernetes.io/projected/53b650ae-6267-45e4-be78-7363c5f6d94b-kube-api-access-d2v7l\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.153674 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.153645    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0fd0630-ea16-413c-a195-ff4cd58d8d21/config-reloader/0.log"
Oct 31 08:14:24.164924 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.164898    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2v7l\" (UniqueName: \"kubernetes.io/projected/53b650ae-6267-45e4-be78-7363c5f6d94b-kube-api-access-d2v7l\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.165048 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.164964    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-sys\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.165110 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.165087    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-lib-modules\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.165220 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.165181    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-sys\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.165433 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.165201    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-proc\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.165575 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.165358    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-proc\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.165575 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.165442    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-lib-modules\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.165751 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.165617    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-podres\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.165751 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.165673    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/53b650ae-6267-45e4-be78-7363c5f6d94b-podres\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.209482 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.209405    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2v7l\" (UniqueName: \"kubernetes.io/projected/53b650ae-6267-45e4-be78-7363c5f6d94b-kube-api-access-d2v7l\") pod \"perf-node-gather-daemonset-prv78\" (UID: \"53b650ae-6267-45e4-be78-7363c5f6d94b\") " pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.256266 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.256235    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.273182 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.273155    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0fd0630-ea16-413c-a195-ff4cd58d8d21/kube-rbac-proxy-web/0.log"
Oct 31 08:14:24.387997 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.387962    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0fd0630-ea16-413c-a195-ff4cd58d8d21/kube-rbac-proxy/0.log"
Oct 31 08:14:24.497436 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.497347    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0fd0630-ea16-413c-a195-ff4cd58d8d21/kube-rbac-proxy-metric/0.log"
Oct 31 08:14:24.554988 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.554952    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"]
Oct 31 08:14:24.558158 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:14:24.558130    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod53b650ae_6267_45e4_be78_7363c5f6d94b.slice/crio-b0f331458531b7ac7fcd1d4f39a576e62e83e80151c8498e82d519e40f4f6068 WatchSource:0}: Error finding container b0f331458531b7ac7fcd1d4f39a576e62e83e80151c8498e82d519e40f4f6068: Status 404 returned error can't find the container with id b0f331458531b7ac7fcd1d4f39a576e62e83e80151c8498e82d519e40f4f6068
Oct 31 08:14:24.605923 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.605894    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0fd0630-ea16-413c-a195-ff4cd58d8d21/prom-label-proxy/0.log"
Oct 31 08:14:24.712085 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.712049    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c0fd0630-ea16-413c-a195-ff4cd58d8d21/init-config-reloader/0.log"
Oct 31 08:14:24.738738 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.738697    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78" event={"ID":"53b650ae-6267-45e4-be78-7363c5f6d94b","Type":"ContainerStarted","Data":"48e070035c231519d9008be7d3adc007c8c1e063d292c40fbbbada5f884b286f"}
Oct 31 08:14:24.738926 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.738747    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78" event={"ID":"53b650ae-6267-45e4-be78-7363c5f6d94b","Type":"ContainerStarted","Data":"b0f331458531b7ac7fcd1d4f39a576e62e83e80151c8498e82d519e40f4f6068"}
Oct 31 08:14:24.738926 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.738828    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:24.777275 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.777173    5829 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78" podStartSLOduration=1.777158224 podStartE2EDuration="1.777158224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 08:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 08:14:24.776222613 +0000 UTC m=+2728.915383204" watchObservedRunningTime="2026-10-31 08:14:24.777158224 +0000 UTC m=+2728.916318866"
Oct 31 08:14:24.957084 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:24.957050    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mrsrf_95b1f652-e473-4b0e-a9da-ac32940f95a8/kube-state-metrics/0.log"
Oct 31 08:14:25.055255 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:25.055165    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mrsrf_95b1f652-e473-4b0e-a9da-ac32940f95a8/kube-rbac-proxy-main/0.log"
Oct 31 08:14:25.162498 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:25.162465    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-mrsrf_95b1f652-e473-4b0e-a9da-ac32940f95a8/kube-rbac-proxy-self/0.log"
Oct 31 08:14:25.271942 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:25.271912    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b67d5b6c9-897nz_c516d07a-8d53-4cae-9f4e-8b07114916b8/metrics-server/0.log"
Oct 31 08:14:25.380019 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:25.379932    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-cgg7t_df7a7365-f2ec-4b65-b8ce-4e7ad72aa7ea/monitoring-plugin/0.log"
Oct 31 08:14:25.822342 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:25.822314    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hbnc6_3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/node-exporter/0.log"
Oct 31 08:14:25.926299 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:25.926262    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hbnc6_3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/kube-rbac-proxy/0.log"
Oct 31 08:14:26.030434 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:26.030400    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hbnc6_3618d95f-af7c-4e0c-8ddd-3a5bbd18e2ee/init-textfile/0.log"
Oct 31 08:14:26.459873 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:26.459842    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pwfwf_658d33ef-d90b-448b-b4d0-e175b77a187f/kube-rbac-proxy-main/0.log"
Oct 31 08:14:26.564790 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:26.564762    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pwfwf_658d33ef-d90b-448b-b4d0-e175b77a187f/kube-rbac-proxy-self/0.log"
Oct 31 08:14:26.672656 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:26.672623    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-pwfwf_658d33ef-d90b-448b-b4d0-e175b77a187f/openshift-state-metrics/0.log"
Oct 31 08:14:26.789885 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:26.789797    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/prometheus/0.log"
Oct 31 08:14:26.894543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:26.894506    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/config-reloader/0.log"
Oct 31 08:14:26.999210 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:26.999175    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/thanos-sidecar/0.log"
Oct 31 08:14:27.104934 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:27.104848    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/kube-rbac-proxy-web/0.log"
Oct 31 08:14:27.210456 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:27.210423    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/kube-rbac-proxy/0.log"
Oct 31 08:14:27.317055 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:27.317021    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/kube-rbac-proxy-thanos/0.log"
Oct 31 08:14:27.422488 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:27.422455    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c4bf6b3f-71ad-4321-9bbf-4d713e4f2110/init-config-reloader/0.log"
Oct 31 08:14:27.855799 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:27.855711    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78cc5444f8-24xwn_2f9af2bb-1fa9-4342-9e95-0ee6b79688e8/telemeter-client/0.log"
Oct 31 08:14:27.960491 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:27.960465    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78cc5444f8-24xwn_2f9af2bb-1fa9-4342-9e95-0ee6b79688e8/reload/0.log"
Oct 31 08:14:28.065835 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:28.065808    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-78cc5444f8-24xwn_2f9af2bb-1fa9-4342-9e95-0ee6b79688e8/kube-rbac-proxy/0.log"
Oct 31 08:14:28.176435 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:28.176403    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8cd777578-wbgnk_f252c1a4-f927-4c73-9521-93f5cfbf2dc4/thanos-query/0.log"
Oct 31 08:14:28.284611 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:28.284580    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8cd777578-wbgnk_f252c1a4-f927-4c73-9521-93f5cfbf2dc4/kube-rbac-proxy-web/0.log"
Oct 31 08:14:28.389906 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:28.389876    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8cd777578-wbgnk_f252c1a4-f927-4c73-9521-93f5cfbf2dc4/kube-rbac-proxy/0.log"
Oct 31 08:14:28.496022 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:28.495941    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8cd777578-wbgnk_f252c1a4-f927-4c73-9521-93f5cfbf2dc4/prom-label-proxy/0.log"
Oct 31 08:14:28.602888 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:28.602858    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8cd777578-wbgnk_f252c1a4-f927-4c73-9521-93f5cfbf2dc4/kube-rbac-proxy-rules/0.log"
Oct 31 08:14:28.709228 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:28.709196    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8cd777578-wbgnk_f252c1a4-f927-4c73-9521-93f5cfbf2dc4/kube-rbac-proxy-metrics/0.log"
Oct 31 08:14:30.754504 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:30.754472    5829 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pl6m9/perf-node-gather-daemonset-prv78"
Oct 31 08:14:35.075095 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:35.075061    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzqlg_457dde04-3e55-4c30-a1c5-d13672f48a4a/dns/0.log"
Oct 31 08:14:35.182112 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:35.182080    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hzqlg_457dde04-3e55-4c30-a1c5-d13672f48a4a/kube-rbac-proxy/0.log"
Oct 31 08:14:35.716130 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:35.716085    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v59hm_ab61d51b-bab9-4ed6-aaa0-1d5e054fefb7/dns-node-resolver/0.log"
Oct 31 08:14:37.039424 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:37.039396    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wbg8z_f206a32f-819d-4e61-9040-790e8c9183ec/node-ca/0.log"
Oct 31 08:14:40.105606 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:40.105572    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cdbcb4849-t7xbb_3992d138-a7d9-42f6-b3b7-6dc9e2cdb478/kube-auth-proxy/0.log"
Oct 31 08:14:40.220520 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:40.220484    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-m8njr_a49dc456-e746-4d51-94ab-124f8abeeb49/istio-proxy/0.log"
Oct 31 08:14:40.432333 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:40.432272    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rw25z_2b71dae3-c259-4a13-af86-ab549638b3d8/payload-pre-processing/0.log"
Oct 31 08:14:40.541177 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:40.541147    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sp8xc_02eee9ac-9709-4b6b-a26c-3fd5ea5e1658/payload-processing/0.log"
Oct 31 08:14:42.057792 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:42.057758    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9lgfj_0cf1ad8c-64aa-4545-87f3-c4f3f9a916c5/serve-healthcheck-canary/0.log"
Oct 31 08:14:43.305215 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:43.305185    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-525tp_b62296f1-d84a-4dd1-8eb0-9f64567af021/kube-rbac-proxy/0.log"
Oct 31 08:14:43.409656 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:43.409622    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-525tp_b62296f1-d84a-4dd1-8eb0-9f64567af021/exporter/0.log"
Oct 31 08:14:43.516116 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:43.516088    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-525tp_b62296f1-d84a-4dd1-8eb0-9f64567af021/extractor/0.log"
Oct 31 08:14:46.515105 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:14:46.515062    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:14:47.210648 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:47.210620    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cdbcb4849-t7xbb_3992d138-a7d9-42f6-b3b7-6dc9e2cdb478/kube-auth-proxy/0.log"
Oct 31 08:14:47.279027 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:47.278993    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-m8njr_a49dc456-e746-4d51-94ab-124f8abeeb49/istio-proxy/0.log"
Oct 31 08:14:47.396358 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:47.396318    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rw25z_2b71dae3-c259-4a13-af86-ab549638b3d8/payload-pre-processing/0.log"
Oct 31 08:14:47.458881 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:47.458845    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sp8xc_02eee9ac-9709-4b6b-a26c-3fd5ea5e1658/payload-processing/0.log"
Oct 31 08:14:47.865036 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:47.865003    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6476fc869f-qhhfq_00c31fe9-8c16-45ad-aa95-01df6db8843d/manager/0.log"
Oct 31 08:14:47.979908 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:47.979873    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6476fc869f-qhhfq_00c31fe9-8c16-45ad-aa95-01df6db8843d/copy-manifests/0.log"
Oct 31 08:14:49.607506 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:49.607470    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-5654d67679-dw56s_60d3af07-65ef-4ee0-b640-f0e8c751440a/manager/0.log"
Oct 31 08:14:49.717844 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:49.717810    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-4wm9m_a491400b-ac46-4ef2-8361-ef6694f91c14/jobset-operator/0.log"
Oct 31 08:14:50.923703 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:14:50.923675    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-66fd44d76c-d9rzk_56f3ce14-6bd4-4aea-8d3b-6bc012bdc7e5/manager/0.log"
Oct 31 08:14:50.995790 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:14:50.995739    5829 controller.go:251] "Failed to update lease" err="Put \"https://a5a808046c859441dad2c6c19fad6d2c-033a58b702821193.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-140-236.ec2.internal?timeout=10s\": context deadline exceeded"
Oct 31 08:15:00.996787 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:15:00.996741    5829 controller.go:251] "Failed to update lease" err="Put \"https://a5a808046c859441dad2c6c19fad6d2c-033a58b702821193.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-140-236.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 08:15:02.837685 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:02.837648    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5jj86_1e3a7ead-4527-476b-9b02-c50e68e80166/kube-multus/0.log"
Oct 31 08:15:03.687611 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:03.687578    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cc6gh_486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/kube-multus-additional-cni-plugins/0.log"
Oct 31 08:15:03.793556 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:03.793527    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cc6gh_486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/egress-router-binary-copy/0.log"
Oct 31 08:15:03.898954 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:03.898921    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cc6gh_486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/cni-plugins/0.log"
Oct 31 08:15:04.005269 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:04.005189    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cc6gh_486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/bond-cni-plugin/0.log"
Oct 31 08:15:04.110981 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:04.110939    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cc6gh_486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/routeoverride-cni/0.log"
Oct 31 08:15:04.216293 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:04.216245    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cc6gh_486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/whereabouts-cni-bincopy/0.log"
Oct 31 08:15:04.321392 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:04.321317    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cc6gh_486ea8a8-d6fe-4a72-b04f-dbd51bc39fdb/whereabouts-cni/0.log"
Oct 31 08:15:05.854117 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:05.854083    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s2r9l_cee91069-0b51-442c-a3c3-71e71302b0e5/network-metrics-daemon/0.log"
Oct 31 08:15:05.948433 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:05.948406    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-s2r9l_cee91069-0b51-442c-a3c3-71e71302b0e5/kube-rbac-proxy/0.log"
Oct 31 08:15:07.561570 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:07.561538    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-controller/0.log"
Oct 31 08:15:07.664715 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:07.664677    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/1.log"
Oct 31 08:15:07.664896 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:07.664720    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovn-acl-logging/0.log"
Oct 31 08:15:07.745977 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:07.745945    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/kube-rbac-proxy-node/0.log"
Oct 31 08:15:07.851584 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:07.851496    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/kube-rbac-proxy-ovn-metrics/0.log"
Oct 31 08:15:07.956330 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:07.956299    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/northd/0.log"
Oct 31 08:15:08.063051 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:08.063020    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/nbdb/0.log"
Oct 31 08:15:08.168199 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:08.168168    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/sbdb/0.log"
Oct 31 08:15:08.389570 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:08.389534    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4g4jz_4fc187c9-2163-4de8-afbb-77f65de777c8/ovnkube-controller/0.log"
Oct 31 08:15:10.997234 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:15:10.997174    5829 controller.go:251] "Failed to update lease" err="Put \"https://a5a808046c859441dad2c6c19fad6d2c-033a58b702821193.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-140-236.ec2.internal?timeout=10s\": context deadline exceeded"
Oct 31 08:15:11.497645 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:11.497608    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cdbcb4849-t7xbb_3992d138-a7d9-42f6-b3b7-6dc9e2cdb478/kube-auth-proxy/0.log"
Oct 31 08:15:11.561794 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:11.561745    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-645cf464d7-m8njr_a49dc456-e746-4d51-94ab-124f8abeeb49/istio-proxy/0.log"
Oct 31 08:15:11.680155 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:11.680124    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-rw25z_2b71dae3-c259-4a13-af86-ab549638b3d8/payload-pre-processing/0.log"
Oct 31 08:15:11.737945 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:11.737919    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-7bb9489676-sp8xc_02eee9ac-9709-4b6b-a26c-3fd5ea5e1658/payload-processing/0.log"
Oct 31 08:15:12.873619 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:12.873585    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zb9td_8856b22a-b791-4ca1-8083-650b391429af/network-check-target-container/0.log"
Oct 31 08:15:15.246987 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:15.246952    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-96gd4_2af799e3-eb79-41f6-be34-411ea2f10326/iptables-alerter/0.log"
Oct 31 08:15:16.124479 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124431    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124479 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124466    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124518    5829 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124520    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124518    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-oe1fmgg4dv9t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124560    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124581    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124609    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124636    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124654    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qk9pc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124663    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124686    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124692    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124707    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124715    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rz6mj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.124717 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124718    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xtmc6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124735    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124749    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124769    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124779    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124771    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124787    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124809    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124818    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:15:16.124784    5829 controller.go:251] "Failed to update lease" err="Put \"https://a5a808046c859441dad2c6c19fad6d2c-033a58b702821193.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-140-236.ec2.internal?timeout=10s\": http2: client connection lost"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124828    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124852    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124906    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124946    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-brgxg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124954    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124975    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.124975    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125008    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125013    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125043    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125081    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125090    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125116    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125119    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125137    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x6ssn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125162    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125169    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125163    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-548vl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125186    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125190    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125211    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125229    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgg5g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125244    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125246    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125253    5829 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125275    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125296    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125309    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125319    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125332    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125353    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125365    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125484    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4qjz7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125504    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125521    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125532    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125540    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125540    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125556    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gjb5j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125562    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jrdcs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125571    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.125543 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125579    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ppvgk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125527    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125582    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-m48c6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125590    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nzlr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125624    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xnqjm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125625    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125640    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vhq6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125650    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125652    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125629    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dql9j\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125659    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wkhqf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125655    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125674    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125686    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125692    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125634    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125706    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125716    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125693    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125729    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125740    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125728    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125757    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125767    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125776    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125767    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125779    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh9bq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125796    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-78uabpe1lkr2i\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125798    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8qb8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125769    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-mghfb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125808    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x4pvw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125815    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125826    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125794    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-must-gather-pl6m9\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125841    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125843    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125847    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125848    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125857    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-58mwv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125873    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125877    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125883    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125897    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125908    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125918    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125926    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125932    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125942    5829 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125907    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xhs6v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125952    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125955    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125929    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125967    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-1lcltrbisfsua\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125952    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m2lq9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125975    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t667t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125988    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125990    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125897    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126001    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125908    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126017    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126026    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-must-gather-pl6m9\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126010    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126045    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126046    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126052    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6pt6b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125975    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7m4f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126071    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126076    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125986    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.125934    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126026    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-fxmlh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126103    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126118    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vrqtd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126122    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126126    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126130    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126106    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126145    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126154    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126029    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126071    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126168    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126173    5829 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126048    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126113    5829 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126158    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f8jfd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126077    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126101    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126202    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cg7nk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126187    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126217    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-must-gather-pl6m9\"/\"default-dockercfg-gs2j4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126221    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126226    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgtls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126208    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-lp6mz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126236    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7l5b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126184    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126246    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-t4ppz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126231    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126199    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126233    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fdnb6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126218    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.127320 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.126189    5829 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Oct 31 08:15:16.512327 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:15:16.510367    5829 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-525tp" containerName="extractor"
Oct 31 08:15:16.878028 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.877943    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-86nzk_836cf52b-b0e8-4e98-a812-21d0cebfaf63/tuned/0.log"
Oct 31 08:15:16.938736 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.938707    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 31 08:15:16.945852 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.945818    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 31 08:15:16.945852 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.945831    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 08:15:16.958156 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.958132    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.959181 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.959158    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 08:15:16.968603 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.968574    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 08:15:16.968603 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.968593    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 08:15:16.968815 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.968593    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 08:15:16.969587 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.969567    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-lp6mz\""
Oct 31 08:15:16.972671 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.972650    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6pt6b\""
Oct 31 08:15:16.980659 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.980641    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hgtls\""
Oct 31 08:15:16.982223 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.982204    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pl6m9\"/\"openshift-service-ca.crt\""
Oct 31 08:15:16.985042 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.985023    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 08:15:16.987093 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.987076    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 08:15:16.988034 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.988016    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 08:15:16.990269 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.990249    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 08:15:16.996072 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:16.996054    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-xnqjm\""
Oct 31 08:15:17.006678 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.006655    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rz6mj\""
Oct 31 08:15:17.010811 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.010793    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 08:15:17.014705 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.014686    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 08:15:17.015317 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.015293    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 08:15:17.020852 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.020836    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 08:15:17.034571 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.034544    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 08:15:17.038512 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.038485    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gjb5j\""
Oct 31 08:15:17.042292 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.042262    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 31 08:15:17.043221 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.043206    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.058775 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.058750    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-brgxg\""
Oct 31 08:15:17.060795 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.060778    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 08:15:17.088164 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.088122    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 31 08:15:17.088396 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.088268    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 08:15:17.088396 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.088340    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.088540 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.088422    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 08:15:17.094194 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.094166    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 08:15:17.105638 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.105613    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 08:15:17.112583 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.112566    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cg7nk\""
Oct 31 08:15:17.115688 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.115673    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 08:15:17.123707 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.123686    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ppvgk\""
Oct 31 08:15:17.126778 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.126757    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wkhqf\""
Oct 31 08:15:17.134157 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.134109    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-m48c6\""
Oct 31 08:15:17.134539 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.134521    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 31 08:15:17.135747 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.135733    5829 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 08:15:17.137835 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.137817    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.143069 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.143052    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.145143 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.145127    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.146879 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.146865    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.152765 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.152746    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 08:15:17.165104 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.165085    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 08:15:17.168273 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.168258    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 08:15:17.175063 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.175045    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.188750 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.188728    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2nzlr\""
Oct 31 08:15:17.193764 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.193745    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 08:15:17.202236 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.202218    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.207322 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.207302    5829 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 08:15:17.209437 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.209421    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x6ssn\""
Oct 31 08:15:17.210341 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.210326    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 31 08:15:17.213269 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.213253    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.217634 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.217618    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-58mwv\""
Oct 31 08:15:17.219693 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.219679    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 08:15:17.228662 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.228643    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 31 08:15:17.239351 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.239326    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 08:15:17.243035 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.243018    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 08:15:17.244208 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.244192    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 31 08:15:17.247431 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.247412    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.247497 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.247488    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fdnb6\""
Oct 31 08:15:17.248321 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.248306    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 08:15:17.250610 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.250598    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.251642 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.251624    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 08:15:17.259365 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.259348    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 08:15:17.261555 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.261541    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.268932 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.268917    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pl6m9\"/\"default-dockercfg-gs2j4\""
Oct 31 08:15:17.273774 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.273758    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qk9pc\""
Oct 31 08:15:17.277920 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.277907    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 31 08:15:17.282253 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.282237    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.286039 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.286023    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 08:15:17.289130 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.289113    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8qb8s\""
Oct 31 08:15:17.290135 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.290120    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 08:15:17.297064 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.297048    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 08:15:17.301190 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.301176    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.305650 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.305636    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 08:15:17.306663 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.306649    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 31 08:15:17.311815 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.311794    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-548vl\""
Oct 31 08:15:17.311969 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.311901    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 08:15:17.316061 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.316045    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.317693 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.317677    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 31 08:15:17.318946 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.318930    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p7l5b\""
Oct 31 08:15:17.320757 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.320739    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 08:15:17.326672 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.326654    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 08:15:17.328226 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.328212    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 08:15:17.329934 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.329916    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.336136 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.336118    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vhq6\""
Oct 31 08:15:17.340657 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.340637    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vrqtd\""
Oct 31 08:15:17.345770 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.345752    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 08:15:17.351479 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.351458    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 08:15:17.357058 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.357040    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.373035 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.373015    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 08:15:17.378390 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.378374    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.380478 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.380460    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7m4f\""
Oct 31 08:15:17.386247 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.386192    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 08:15:17.395853 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.395835    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mghfb\""
Oct 31 08:15:17.398477 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.398460    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-m2lq9\""
Oct 31 08:15:17.400490 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.400474    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-f8jfd\""
Oct 31 08:15:17.413819 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.413801    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-78uabpe1lkr2i\""
Oct 31 08:15:17.421414 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.421399    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-xhs6v\""
Oct 31 08:15:17.424632 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.424616    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 08:15:17.435100 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.435080    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 31 08:15:17.435692 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.435678    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fh9bq\""
Oct 31 08:15:17.445348 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.445329    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 08:15:17.451018 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.451002    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vgg5g\""
Oct 31 08:15:17.459533 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.459514    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.460634 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.460618    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.463349 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.463328    5829 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 08:15:17.463349 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.463340    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.463499 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.463329    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 08:15:17.465410 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.465397    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 08:15:17.470628 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.470609    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 08:15:17.483914 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.483887    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.492187 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.492163    5829 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 08:15:17.494861 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.494841    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 08:15:17.495975 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.495956    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 08:15:17.498101 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.498082    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.499099 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.499071    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 08:15:17.500379 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.500363    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 08:15:17.511323 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.511301    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 08:15:17.517863 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.517844    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1lcltrbisfsua\""
Oct 31 08:15:17.522441 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.522421    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 08:15:17.525936 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.525916    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xtmc6\""
Oct 31 08:15:17.526971 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.526953    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 08:15:17.528926 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.528910    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 08:15:17.534669 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.534649    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 08:15:17.536673 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.536653    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-fxmlh\""
Oct 31 08:15:17.536856 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.536840    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 08:15:17.551317 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.551267    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.553266 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.553247    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.557495 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.557473    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.560539 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.560516    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pl6m9\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.563427 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.563408    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 08:15:17.567556 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.567536    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 08:15:17.574266 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.574246    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 31 08:15:17.581973 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.581957    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4qjz7\""
Oct 31 08:15:17.583966 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.583948    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-t4ppz\""
Oct 31 08:15:17.586569 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.586556    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.597300 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.597268    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.599853 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.599836    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-oe1fmgg4dv9t\""
Oct 31 08:15:17.602062 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.602039    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 08:15:17.602136 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.602080    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t667t\""
Oct 31 08:15:17.602960 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.602944    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.603958 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.603945    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 08:15:17.606026 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.606012    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 08:15:17.606335 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.606320    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 31 08:15:17.606395 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.606341    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dql9j\""
Oct 31 08:15:17.614019 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.614001    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.616999 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.616984    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 08:15:17.618458 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.618443    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.622752 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.622730    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.624743 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.624726    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 08:15:17.626865 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.626847    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 08:15:17.633089 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.633068    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jrdcs\""
Oct 31 08:15:17.633840 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.633816    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 08:15:17.633891 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.633861    5829 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 08:15:17.634089 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.634075    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.637096 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.637079    5829 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"]
Oct 31 08:15:17.646119 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.646098    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"
Oct 31 08:15:17.656469 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.656450    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-x4pvw\""
Oct 31 08:15:17.658646 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.658628    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.658728 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.658694    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.658728 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.658714    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-phfw6\""
Oct 31 08:15:17.671076 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.671055    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 08:15:17.672116 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.672101    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.679781 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.679742    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 08:15:17.683173 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.683155    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 08:15:17.686268 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.686252    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 08:15:17.696755 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.696731    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lc45\" (UniqueName: \"kubernetes.io/projected/aa6d709a-7337-4506-a7db-0153fe6777cb-kube-api-access-5lc45\") pod \"maas-api-key-cleanup-29890575-nnttj\" (UID: \"aa6d709a-7337-4506-a7db-0153fe6777cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"
Oct 31 08:15:17.696837 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.696804    5829 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6d709a-7337-4506-a7db-0153fe6777cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890575-nnttj\" (UID: \"aa6d709a-7337-4506-a7db-0153fe6777cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"
Oct 31 08:15:17.714402 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.714374    5829 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 08:15:17.721011 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.720986    5829 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 08:15:17.798341 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.798269    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lc45\" (UniqueName: \"kubernetes.io/projected/aa6d709a-7337-4506-a7db-0153fe6777cb-kube-api-access-5lc45\") pod \"maas-api-key-cleanup-29890575-nnttj\" (UID: \"aa6d709a-7337-4506-a7db-0153fe6777cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"
Oct 31 08:15:17.798490 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.798407    5829 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6d709a-7337-4506-a7db-0153fe6777cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890575-nnttj\" (UID: \"aa6d709a-7337-4506-a7db-0153fe6777cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"
Oct 31 08:15:17.799072 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.799052    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa6d709a-7337-4506-a7db-0153fe6777cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890575-nnttj\" (UID: \"aa6d709a-7337-4506-a7db-0153fe6777cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"
Oct 31 08:15:17.815301 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.815235    5829 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lc45\" (UniqueName: \"kubernetes.io/projected/aa6d709a-7337-4506-a7db-0153fe6777cb-kube-api-access-5lc45\") pod \"maas-api-key-cleanup-29890575-nnttj\" (UID: \"aa6d709a-7337-4506-a7db-0153fe6777cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"
Oct 31 08:15:17.958570 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:17.958483    5829 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"
Oct 31 08:15:18.256728 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:18.256534    5829 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj"]
Oct 31 08:15:18.259129 ip-10-0-140-236 kubenswrapper[5829]: W1031 08:15:18.259099    5829 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6d709a_7337_4506_a7db_0153fe6777cb.slice/crio-37d75123e5f7cfb57c1bead77c51b6c54789b95b11aefa5efc9d13ad7fbfbed0 WatchSource:0}: Error finding container 37d75123e5f7cfb57c1bead77c51b6c54789b95b11aefa5efc9d13ad7fbfbed0: Status 404 returned error can't find the container with id 37d75123e5f7cfb57c1bead77c51b6c54789b95b11aefa5efc9d13ad7fbfbed0
Oct 31 08:15:19.030260 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:19.030220    5829 generic.go:356] "Generic (PLEG): container finished" podID="aa6d709a-7337-4506-a7db-0153fe6777cb" containerID="70f829675f5d2b52287fdd1a4f5dd4325423734eac976a8a85e8bc7c0cf7b0df" exitCode=60
Oct 31 08:15:19.030650 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:19.030307    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj" event={"ID":"aa6d709a-7337-4506-a7db-0153fe6777cb","Type":"ContainerDied","Data":"70f829675f5d2b52287fdd1a4f5dd4325423734eac976a8a85e8bc7c0cf7b0df"}
Oct 31 08:15:19.030650 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:19.030365    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj" event={"ID":"aa6d709a-7337-4506-a7db-0153fe6777cb","Type":"ContainerStarted","Data":"37d75123e5f7cfb57c1bead77c51b6c54789b95b11aefa5efc9d13ad7fbfbed0"}
Oct 31 08:15:19.030650 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:19.030592    5829 scope.go:122] "RemoveContainer" containerID="70f829675f5d2b52287fdd1a4f5dd4325423734eac976a8a85e8bc7c0cf7b0df"
Oct 31 08:15:20.038500 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:20.038415    5829 generic.go:356] "Generic (PLEG): container finished" podID="aa6d709a-7337-4506-a7db-0153fe6777cb" containerID="c65282aa5c54613908bbafca02ae458e6be5a7e9f0a61c77bd28255dbcc20f02" exitCode=60
Oct 31 08:15:20.038939 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:20.038497    5829 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj" event={"ID":"aa6d709a-7337-4506-a7db-0153fe6777cb","Type":"ContainerDied","Data":"c65282aa5c54613908bbafca02ae458e6be5a7e9f0a61c77bd28255dbcc20f02"}
Oct 31 08:15:20.038939 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:20.038565    5829 scope.go:122] "RemoveContainer" containerID="70f829675f5d2b52287fdd1a4f5dd4325423734eac976a8a85e8bc7c0cf7b0df"
Oct 31 08:15:20.038939 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:20.038849    5829 scope.go:122] "RemoveContainer" containerID="c65282aa5c54613908bbafca02ae458e6be5a7e9f0a61c77bd28255dbcc20f02"
Oct 31 08:15:20.039165 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:15:20.039142    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890575-nnttj_odh-ai-gateway-infra(aa6d709a-7337-4506-a7db-0153fe6777cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj" podUID="aa6d709a-7337-4506-a7db-0153fe6777cb"
Oct 31 08:15:21.046041 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:21.046007    5829 scope.go:122] "RemoveContainer" containerID="c65282aa5c54613908bbafca02ae458e6be5a7e9f0a61c77bd28255dbcc20f02"
Oct 31 08:15:21.046496 ip-10-0-140-236 kubenswrapper[5829]: E1031 08:15:21.046255    5829 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890575-nnttj_odh-ai-gateway-infra(aa6d709a-7337-4506-a7db-0153fe6777cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890575-nnttj" podUID="aa6d709a-7337-4506-a7db-0153fe6777cb"
Oct 31 08:15:24.970697 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:24.970664    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-bcbfl_db612775-57b2-405c-a403-8c78f173d018/csi-driver/0.log"
Oct 31 08:15:25.074961 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:25.074928    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-bcbfl_db612775-57b2-405c-a403-8c78f173d018/csi-node-driver-registrar/0.log"
Oct 31 08:15:25.179654 ip-10-0-140-236 kubenswrapper[5829]: I1031 08:15:25.179617    5829 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-bcbfl_db612775-57b2-405c-a403-8c78f173d018/csi-liveness-probe/0.log"